Support multiple JamfUpload Processors running concurrently

[MLBZ521]

Support concurrency in JamfUpload by:

• Having each session will use it's own Bear Token
• Having each separate recipe run save it's session data to a unique directory and files
• Data returned from each API query will be written to unique files (e.g. curl --output)
• Each each session is self contained, but will use the same cookie's, etc. to ensure (as much as we can) the session continues to talk to the same JPS node

This method will prevent and not allow any chance for different autopkg runs/session to invalidate the others Bear Token or overwrite their curl --output files, nor anything else.

I was getting random failures, ~20 recipes out of ~100, with each autopkg run session, which are run multiple times a day (every four hours). Each execution would have random different recipes fail. Almost all failures appeared to be random API errors, which were hard to track down, as each recipe run was overwriting the results from another recipe run. The method I'm using to run my recipes uses concurrency (multiple autopkg run <recipe_id>s are being ran at a time.

Before implementing this change, I decreased the concurrency down to two and was still experiencing this behavior, but after implementing the changes in this commit, I've been able to increase the concurrency to six and have not received one single error during a JamfUpload step for over a week now.

[grahampugh]

Hi Zack, this is interesting. I never even considered that AutoPkg could be concurrent at all - certainly not a workflow I've tried from the same device. I notice that you have not altered the header and cookie files to point to the temporary directory, so these will continue to be used by each session. I'm not sure if you intended that, because the header file is read to figure out whether a request was successful, and the cookie file could be being manipulated differently by different runs (e.g. where the JSS_URL has been overridden). How do you think it would work if these were also pointed to the temp directory to entirely self-contain each run? Obviously that would slow down the first request as it would have to obtain a new token (and I don't know if issuing a new token automatically invalidates an old one).

[MLBZ521]

It can, or at least, I have been running it for a bit pretty successfully.

I'm not the first to do/try it. @Groob has autopkgd, which is a Go implementation (Blog'd on it here: Sysadmin productivity with Go). Though my implementation, Pkgbot, is different.

The cookie and header files are altered (but nothing changed in those specific lines (L291 & L293)) -- I updated make_tmp_dir() to create a temporary directory: https://github.com/grahampugh/jamf-upload/blob/3b60afd3a51350cc91e619d28f696e6df332e090/JamfUploaderProcessors/JamfUploaderLib/JamfUploaderBase.py#L290-L293

https://github.com/grahampugh/jamf-upload/blob/3b60afd3a51350cc91e619d28f696e6df332e090/JamfUploaderProcessors/JamfUploaderLib/JamfUploaderBase.py#L122-L127

Yes, it does have to obtain a new token, but the overhead isn't noticeable -- I tried with a shared token, but one process/run would randomly (eventually) invalidate the token another process/run was in the middle of using.

Issuing a Bearer Token does not invalidate a previous Bearer Token. You can have an infinite number of tokens (as far as I know at least). To test, go to the https://my.jps.org:8443/api/doc/#/api-authentication/post_v1_auth_token endpoint and execute the sample endpoint. Copy the token and then execute it again. Then do a simple GET against any endpoint, e.g.

curl -X GET "https://my.jps.org:8443/api/v1/jamf-pro-version" -H "accept: application/json" -H "Authorization: Bearer <token>"

Until the token expires or you explicitly invalidate the token, it's still valid -- otherwise, people would have all kinds of huge issues for those that are using hard coded API credentials in scripts on devices (not that I condone/recommend doing so).