URLDownloaderPython unable to be used as a SharedProcessor without your autopkg fork

[discentem]

Describe the bug URLDownloaderPython cannot be referenced a SharedProcessor (i.e. com.github.jgstew.SharedProcessors/URLDownloaderPython) from the current autopkg release version because the processor code assumes URLDownloaderPython is in autopkglib: https://github.com/jgstew/jgstew-recipes/blob/main/SharedProcessors/URLDownloaderPython.py#L10

...
WARNING: C:\path\to\RecipeRepos\com.github.jgstew.jgstew-recipes\SharedProcessors\URLDownloaderPython.py: No module named 'autopkglib.URLDownloaderPython'
...

To Reproduce

1. autopkg repo-add --prefs=autopkg_prefs.json https://github.com/jgstew/jgstew-recipes.git
2. Try to reference URLDownloaderPython in a recipe using stub recipe:

Process:
  - Processor: com.github.jgstew.SharedProcessors/URLDownloaderPython

Desktop (please complete the following information):

• OS: Windows

Additional context I think the fix would be to have the SharedProcessor actually contain all of the implementation code, which is currently only here in full.

[jgstew]

autopkglib.URLDownloaderPython is only found within the Dev branch of AutoPkg itself. I also have some enhancements to it that have not been integrated yet within my fork of AutoPkg.

I am currently running, developing recipes, and testing AutoPkg on MacOS, Ubuntu, and Windows. Not all of the processors work on all platforms, but that is the end goal in most cases for me.

URLDownloaderPython was designed to address crossplatform issues with URLDownloader, particularly on Ubuntu Docker Containers, Ubuntu WSL, and Windows, as well as add lots of enhancements. That said it is still new and there are a few rough edges and improvements to make. That said, I'm using URLDownloaderPython very extensively for a lot of recipes, so from that perspective, it is very well tested, especially on Windows and Ubuntu.

All that said, I have considered switching back to having all of the implementation code within com.github.jgstew.SharedProcessors/URLDownloaderPython as the forward looking code path, while autopkglib.URLDownloaderPython being the more stable version of the same.

For now, the newest code for URLDownloaderPython is here: https://github.com/jgstew/autopkg/blob/dev/Code/autopkglib/URLDownloaderPython.py

The version integrated into AutoPkg itself is currently here: https://github.com/autopkg/autopkg/blob/dev/Code/autopkglib/URLDownloaderPython.py

If you are running AutoPkg on Windows, then I highly recommend the Dev branch to do so since it is still slightly experimental. See the check setup script here: https://github.com/jgstew/jgstew-recipes/blob/main/check_setup_win.bat

The check-setup script does fully enumerate the setup required to run AutoPkg on Windows, but also includes stuff that is more specific to running jgstew-recipes but not AutoPkg on Windows generically. I should perhaps split it up.

[jgstew]

I just saw this: https://github.com/discentem/discentem-recipes/blob/main/SharedProcessors/WindowsSignatureVerifier.py

That is definitely useful. I was actually hoping to make a cross platform version of the same using https://github.com/ralphje/signify

[discentem]

Hey @jgstew, thanks for the very detailed reply. In short, I think you are saying I should just use Autopkg dev and that's fair if that's your decision. I figured someone could get value out of the processor from Autopkg master though 🤷

All that said, I have considered switching back to having all of the implementation code within com.github.jgstew.SharedProcessors/URLDownloaderPython as the forward looking code path, while autopkglib.URLDownloaderPython being the more stable version of the same.

I see you added a bug label 👍 but i'm not actually sure what you are deciding haha. Can you clarify? I'm a little unclear on "I have considered switching..."

And yes i am working on windows things as well, albeit slowly and in my free time: https://github.com/discentem/discentem-recipes/blob/main/windows-recipes/bootstrap_autopkg.ps1

I was going to work on a script to bootstrap signtool next: https://stackoverflow.com/questions/31869552/how-to-install-signtool-exe-for-windows-10

[discentem]

https://github.com/ralphje/signify

I unfortunately could not figure out how to make this work, I did try. Excited to see if you can make heads or tails of it. It seemed unable to parse release of https://github.com/microsoft/terminal (which are msixbundles) despite the fact that Get-AuthenticodeSignature and Signtool both worked as expected.

[jgstew]

I was going to work on a script to bootstrap signtool next

There is also this tool, which may be more cross platform than signtool: https://github.com/mtrojnar/osslsigncode

It also seems like this should be a solved problem in .NET core / PowerShell core? ... or not?

I unfortunately could not figure out how to make this work, I did try.

That is a bummer. I'll see if I get any further, but I don't know I would disqualify it completely as a partial option if it can work for most things other than MSIX bundles. I wonder if MSIX bundles are not technically PEFiles and that is why they don't work?

---

One thing I'm looking at doing is having processors stack on top of each other that do the same thing. Like getting the ProductVersion from an MSI. Use python's windows only MSILib module in a windows only processor that just does nothing on non-windows, then have a non-Windows processor that uses a different binary to get the same thing but does nothing on Windows. My goal is for the majority of my recipes to run on any platform regardless of which OS they create content for.

I think you are saying I should just use Autopkg dev and that's fair if that's your decision. I figured someone could get value out of the processor from Autopkg master though

I agree, but Windows support is not technically a part of AutoPkg master, and the goal is for URLDownloaderPython to be integrated into AutoPkg master from dev, so then it is no longer an issue. That said, I do think it makes sense to put the newest branch of code back into the shared processor.

but i'm not actually sure what you are deciding haha. Can you clarify? I'm a little unclear on "I have considered switching..."

I am unable to clarify because I am uncertain myself, though I think I've been convinced to address this and put the code back into the shared processor.

[discentem]

That said, I do think it makes sense to put the newest branch of code back into the shared processor.

This clarifies it for me, thank you!

There is also this thing from Trail of Bits for verifying binaries but there are some caveats. If this is a solved problem in Powershell core 🎉 i think that's best.

[jgstew]

This should be addressed now. I haven't tested it yet.

[discentem]

Thanks! I'd love to keep chatting with you on Slack about Autopkg Windows stuff, if you are still active on there. 😄

[jgstew]

I just added the MacAdmins slack back to my system, so I am available there now. I guess I will close this issue, but feel free to open a bug issue if there are any problems when testing.