Closed MLBZ521 closed 1 year ago
Hi Zack, this is interesting. I never even considered that AutoPkg could be concurrent at all - certainly not a workflow I've tried from the same device. I notice that you have not altered the header and cookie files to point to the temporary directory, so these will continue to be used by each session. I'm not sure if you intended that, because the header file is read to figure out whether a request was successful, and the cookie file could be being manipulated differently by different runs (e.g. where the JSS_URL has been overridden). How do you think it would work if these were also pointed to the temp directory to entirely self-contain each run? Obviously that would slow down the first request as it would have to obtain a new token (and I don't know if issuing a new token automatically invalidates an old one).
It can, or at least, I have been running it for a bit pretty successfully.
I'm not the first to do/try it. @Groob has autopkgd, which is a Go implementation (Blog'd on it here: Sysadmin productivity with Go). Though my implementation, Pkgbot, is different.
The cookie and header files are altered (but nothing changed in those specific lines (L291 & L293)) -- I updated make_tmp_dir()
to create a temporary directory:
https://github.com/grahampugh/jamf-upload/blob/3b60afd3a51350cc91e619d28f696e6df332e090/JamfUploaderProcessors/JamfUploaderLib/JamfUploaderBase.py#L290-L293
Yes, it does have to obtain a new token, but the overhead isn't noticeable -- I tried with a shared token, but one process/run would randomly (eventually) invalidate the token another process/run was in the middle of using.
Issuing a Bearer Token does not invalidate a previous Bearer Token. You can have an infinite number of tokens (as far as I know at least). To test, go to the https://my.jps.org:8443/api/doc/#/api-authentication/post_v1_auth_token
endpoint and execute the sample endpoint. Copy the token and then execute it again. Then do a simple GET
against any endpoint, e.g.
curl -X GET "https://my.jps.org:8443/api/v1/jamf-pro-version" -H "accept: application/json" -H "Authorization: Bearer <token>"
Until the token expires or you explicitly invalidate the token, it's still valid -- otherwise, people would have all kinds of huge issues for those that are using hard coded API credentials in scripts on devices (not that I condone/recommend doing so).
Support concurrency in JamfUpload by:
curl --output
)This method will prevent and not allow any chance for different
autopkg run
s/session to invalidate the others Bear Token or overwrite theircurl --output
files, nor anything else.I was getting random failures, ~20 recipes out of ~100, with each
autopkg run
session, which are run multiple times a day (every four hours). Each execution would have random different recipes fail. Almost all failures appeared to be random API errors, which were hard to track down, as each recipe run was overwriting the results from another recipe run. The method I'm using to run my recipes uses concurrency (multipleautopkg run <recipe_id>
s are being ran at a time.Before implementing this change, I decreased the concurrency down to two and was still experiencing this behavior, but after implementing the changes in this commit, I've been able to increase the concurrency to six and have not received one single error during a JamfUpload step for over a week now.