[1.0.2b2] Cannot upload packages to CDP (S3)

smithjw commented 6 years ago

Tried out the latest 1.0.1 beta but I'm still running into constant errors. I've included some output from autopkg runs which I'm hoping will help track the issue down.

Before running this I did the following:

Deleted the Policy from Jamf
Removed the latest Atom pkg from S3
Removed the latest Atom pkg from my File Share Distribution Point

The following is a breakdown of what occurred after each run

Run 1:

Atom is downloaded and packaged
Atom pkg record is created in the Jamf server

Run 2:

Atom policy is created on the Jamf Server
Atom pkg is copied to the Local file share
Atom pkg is uploaded to S3 but the file is broken (only 70MB vs 132MB)

Run 3: (Before running I delete the pkg record from Jamf which also removes the broken file from S3 and from the Policy)

Package record is created in Jamf
No File is uploaded to S3
No Package is attached to the Atom Policy

JSSImporter error log.txt

cranetech commented 6 years ago

Same exact issue.

macataxis commented 6 years ago

Same here. Files on S3 turns up ~half size.

sheagcraig commented 6 years ago

Thanks for the status folks.

If @mosen wants to chime in with suggestions, awesome!

I'm not in a situation to solve this problem (I don't have Jamf Pro or a CDP) nor time at work to do much research. If someone can send me some POC code showing how to upload a file to the CDP I can code it up, even if it's a curl command instead of python. But I'm going to need some help on the research end.

mosen commented 6 years ago

Earmarked some time on the weekend to look at this.

macataxis commented 6 years ago

Tnx. For your information. We rolled back to 0.5.1 and got it working again.

MScottBlake commented 6 years ago

I installed 1.0 and nothing would upload, then I went to the pre-release, and everything uploaded, every time. I downgraded back to 0.5.1 like @macataxis.

If you need any help troubleshooting, let me know; I have a CDP.

sheagcraig commented 6 years ago

Based on the comments that 0.5.1 works fine, this is almost certainly just an omission in the curl arguments or a mistake in the curl arguments. 0.5.1 is doing the request with python requests, and 1.0.x is using curl.

If someone can successfully manually curl a package to their CDP and post up the arguments required for 100% success, we could compare those args to the ones python-jss is generating to see what the discrepancy is.

sheagcraig commented 6 years ago

@MScottBlake this is something I bet you could figure out pretty easily!

MScottBlake commented 6 years ago

Okay @sheagcraig and @mosen, I may be on to something here. I was able to manually upload a file to a CDP using only cURL. Here are my results.

Command:

curl -u user:pass -X POST https://jss.domain.com:8443/dbfileupload -H 'DESTINATION: 0' -H 'OBJECT_ID: -1' -H 'FILE_TYPE: 0' -H 'FILE_NAME: VMwareTools-10.2.0.pkg' -T '/path/to/VMwareTools-10.2.0.pkg'

Result:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><uploadResponse><id>4767</id><md5>1e3fff43b2f907f9ed2e6f1c4495d6fa</md5><message></message><receivedSize>2605597</receivedSize><successful>true</successful></uploadResponse>

Notice the receivedSize is 2605597. This is the correct file size. The file appears in my S3 bucket at 2.5MB as I would expect. However, if I change the -T to a -d, I get approximately 50% of the file and S3 shows a size of 1.3MB. This is what I experienced and what I've seen others report.

Command:

curl -u user:pass -X POST https://jss.domain.com:8443/dbfileupload -H 'DESTINATION: 0' -H 'OBJECT_ID: -1' -H 'FILE_TYPE: 0' -H 'FILE_NAME: VMwareTools-10.2.0.pkg' -d '@/path/to/VMwareTools-10.2.0.pkg'

Result:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><uploadResponse><id>4768</id><md5>e0787449fcbbb2a39e38dc6c0f143853</md5><message></message><receivedSize>1399246</receivedSize><successful>true</successful></uploadResponse>

I don't know what parameters are being used, but I suspect it is the equivalent of the -d parameter in cURL.

Unfortunately, I do not have the ability to test the uploaded files at this time and I will be offline until the weekend. Hopefully this information helps in some capacity.

elvisizer commented 6 years ago

@sheagcraig @mosen running into this as well, also getting the same results with -T and -d on the curl commands as @MScottBlake : -T works. Anything else I can do to help with this?

sheagcraig commented 6 years ago

@elvisizer

I've got a beta on the releases that uses @MScottBlake suggestions to do the curl. I've heard from one person that it works.

That person is convincingly authoritative, but I'd love for another tester to confirm.

macataxis commented 6 years ago

Seems to be working. TNX!

11 juni 2018 kl. 16:55 skrev Shea Craig notifications@github.com<mailto:notifications@github.com>:

@elvisizerhttps://github.com/elvisizer

I've got a beta on the releases that uses @MScottBlakehttps://github.com/MScottBlake suggestions to do the curl. I've heard from one person that it works.

That person is convincingly authoritative, but I'd love for another tester to confirm.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/sheagcraig/JSSImporter/issues/128#issuecomment-396272885, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Af5-8IkC4dT87H6iGUtHR27L22KTm6o1ks5t7oTegaJpZM4R6WXv.

sktaylortrash commented 6 years ago

was this last fix meant only for S3 CDP installs or should it potentially have worked for jamfcloud installations as well?

elvisizer commented 6 years ago

Beta 3 worked with my S3 bucket, but I did still have to modify the jssobject.py file by adding import time on line 27, and time.sleep(60) on line 551.

MatGriffin commented 6 years ago

I'm also seeing half size pkgs on AWS CDP.

What's the best fix here, currently I'm using: macOS 10.13.6 Autopkgr 1.4.2 JSSImporter 1.0.2b2 - tried latest but got more errors Jamf 10.7.1

looking forward to the direct AWS option :)

mosen commented 6 years ago

I believe that somewhere between Jamf Pro 10.5.0 and 10.7.1 the implementation of /dbfileupload was changed. This is the URL that JSSImporter has always used to upload packages for CDP, JCDS and JDS.

Seeing as this option seems deprecated/end of life and we don't have an official API, I would like to offer direct upload options for AWS S3 CDP and JCDS (which already exists).

The old distribution point "CDP" will remain for people running older versions of JAMF Pro.

grahampugh commented 4 years ago

Closing this as it relates to the old version that used curl. If anyone is still seeing the issue, comment here and I’ll reopen.

jssimporter / JSSImporter

[1.0.2b2] Cannot upload packages to CDP (S3) #128