Autopkgr stops running after ~20 recipes

[novaksam]

I'm not sure if it's just me, but it seems like autopkgr stops running after about 20 recipes. I initially thought it might be an issue with particular recipes (GoogleDrive, Handbrake, etc) but it appears that it has some sort of issue processing more than 20.

I can provide the list of recipes I'm using if it would be helpful in debugging.

[eahrold]

Yeah, go ahead and post the contents of your ~/Library/Application Support/AutoPkgr/recipe_list.txt. And let us know on which recipe it hangs.

And using Activity Monitor, could you see if you have any excessive memory/cpu usage at the time it stops.

[novaksam]

local.pkg.CasperSuite local.jss.AdobeAIR local.jss.AdobeAcrobatProXUpdate local.jss.AdobeDigitalEditions local.jss.AdobeFlash local.jss.AdobeReader local.jss.AdobeShockwavePlayer local.jss.Audacity local.jss.Compressor local.jss.Dropbox local.jss.Evernote local.jss.FinalCutPro local.jss.Firefox local.jss.GarageBand local.jss.GoogleChrome local.jss.Github local.jss.Keynote < Locking up local.jss.MicrosoftRemoteDesktop local.jss.Motion local.jss.Numbers local.jss.OracleJava8 local.jss.OracleJava8JDK local.jss.Pages local.jss.Processing local.jss.SassafrasK2Client local.jss.Skype local.jss.Silverlight local.jss.TextWrangler local.jss.TheUnarchiver local.jss.VLC local.jss.VMwareFusion local.jss.Wireshark local.jss.XCode local.jss.XQuartz local.jss.iMovie local.jss.iPhoto local.jss.MicrosoftLync local.jss.AndroidStudio local.jss.SuspiciousPackage local.jss.MSOffice2011Updates local.jss.OpenOffice local.jss.Handbrake local.jss.GoogleDrive

https://gist.github.com/novaksam/c9b5a5e9c442108d19f9

[novaksam]

No spikes or anything like that, it seems to just stop processing after a certain point.

[eahrold]

@homebysix, I don't have have the the AppStore apps like Keynote, Pages, and Numbers. Do you have an env where you could test this list?

[homebysix]

I routinely run 34 recipes on one of my Macs (including .download, .pkg, and .jss recipes) without trouble. @novaksam, does it always stop processing after the same "certain point" or does that point change unpredictably?

I haven't used the AppStoreApp recipes yet, but I'll set up a test environment to play with those.

[novaksam]

It seems to stop on certain ones. I've noticed it stopping on GoogleDrive, Handbrake and now Keynote.

[homebysix]

@novaksam, what happens if you run autopkg run GoogleDrive.jss -v in Terminal? Does it succeed?

Same with Handbrake and Keynote?

[novaksam]

Those work fine, I have a script that runs the recipes in a for loop and it never seems to bail out.

I'll try running them verbose and seeing what it says.

[novaksam]

https://gist.github.com/novaksam/fa3931a2190de275b147

[novaksam]

Perhaps there's an issue running processors for some of these other recipes?

[homebysix]

This part is interesting:

Unmount successful for /Volumes/CasperShare
Unmount successful for /Volumes/CasperShare-1
Unmount successful for /Volumes/CasperShare-1-2
Unmount successful for /Volumes/CasperShare-1-2-3

Do you know what's going on there?

[novaksam]

It's unmounting the JSS shares.

[homebysix]

Are the shares really called CasperShare, CasperShare-1, CasperShare-1-2, and CasperShare-1-2-3, or is that auto-incrementation caused by duplicate share names?

[novaksam]

Yeah it's auto incrementing.

[novaksam]

I just ran the same autopkg command that autoPkgr uses without issue.

/usr/bin/python /usr/local/bin/autopkg run --recipe-list "/Users/adm/Library/Application Support/AutoPkgr/recipe_list.txt" --report-plist /var/folders/33/hvd2nwbx54z0t7gqb4k_f4sc0000gn/T/com.lindegroup.AutoPkgr/20150422080739 -v

[eahrold]

@novaksam, There is some indication in the /usr/bin/sample you took that could indicate an underlying buffer getting filled up from STDOUT of the AutoPkg that AutoPkgr uses to process UI updates. I'm looking into this a bit more and will keep you posted.

[eahrold]

@novaksam, I've played around with the I/O a little bit, mind giving this build a test... https://dl.dropboxusercontent.com/u/120270/AutoPkgr/AutoPkgr_issue%23334.zip

[novaksam]

https://gist.github.com/novaksam/07e9112f7fabb43b2c6e

[novaksam]

https://gist.github.com/novaksam/9eba2ad0af28d888e906

[eahrold]

@novaksam Looks like there still is some sort of buffer issue. This is what leads me to this __write_nocancel (in libsystem_kernel.dylib) + 10 [0x7fff8754e97a]

This may help...

defaults write com.github.autopkg JSS_SUPPRESS_WARNINGS -bool True

I may put up some more builds for you to test if that's OK.

Thanks, Eldon

[novaksam]

I'm running it now with that new default in place and I'll report back if it stalls.

[eahrold]

@novaksam Also could you put the samples you take in a Gist and just post the link, navigating this issue is getting a bit tricky.

Thanks...

[novaksam]

Can do, I'll update the past posts into gists.

[novaksam]

It completed the whole list without stalling; I'm guessing you're right about the buffer filling up.

[eahrold]

Wow! That's actually somewhat unexpected, I'll have to look into this more. Run a few more times periodically today and let us know either way whether things stall out again.

Thanks, Eldon

[eahrold]

Also are you using the custom build I put up? If so would you mind trying with the Release version?

[novaksam]

Sure, I'll try the release.

[novaksam]

Yeah it works fine with the latest release too.

[eahrold]

@novaksam, I just rewrote the error handler to progressively collect the stderr. https://www.dropbox.com/s/i5cnl6pb8ct378m/AutoPkgr-ErrorHandler.zip?dl=0

Would you mind turning back on the JSS_WARNINGS and see if this new build fixes the original issue? defaults write com.github.autopkg JSS_SUPPRESS_WARNINGS -bool False

[novaksam]

It looks like that re-write did the trick. It's going through all the recipes without stalling.

[eahrold]

Excellent! Thanks for all your help working through this.

[novaksam]

No problem! Glad we got it figured out.

I'll close the issue now.