AutoPkg Dev branch does not run recipes in current working directory

[jgstew]

THIS IS ONLY INTENDED FOR AUTOPKG BETAS.

Describe the problem In the Dev branch, AutoPkg will build a recipe map and use it to look up recipe identifiers to full paths to the recipes. This is really excellent and speeds up AutoPkg executions by a TON when many recipes are involved.

That said, AutoPkg will NOT find all recipes by identifier as given in the RECIPE_SEARCH_DIRS because it does not automatically build the map to include recipes in the current working directory recipes it seems.

I was looking into updating me map file myself to include the missing items, but it does seem that long term this is something that needs to work.

Part of the problem, I'm guessing, is that the current working directory can be different at different times, so the map for it might not make sense to be a global config.

I have not tested this problem anywhere but Win10 at the moment.

This problem is specifically called out in the error message: We are not currently searching on-disk for recipes; if you intended to run a specific file not in the recipe map, please pass in an absolute file path instead.

The issue is, if I do call out an absolute or relative path to a recipe that does not have a parent recipe, then everything works fine. The issue becomes if I do try to run a recipe that has a parent recipe, that parent recipe is given by identifier, and autopkg fails to find it.

Preferences contents

python ..\autopkg\Code\autopkg info
WARNING: Library 'xattr' unavailable. Defining no-op implementation.
WARNING: Failed 'from Foundation import NSDictionary' in autopkglib.MunkiInstallsItemsCreator
WARNING: Failed 'from Foundation import CFPreferencesCopyAppValue' in autopkglib.MunkiSetDefaultCatalog
WARNING: Failed 'from Foundation import NSPredicate' in autopkglib.StopProcessingIf
Current preferences:
{'RECIPE_REPOS': {'C:\\Users\\james\\Library\\AutoPkg\\RecipeRepos\\com.github.autopkg.hansen-m-recipes': {'URL': 'https://github.com/autopkg/hansen-m-recipes'},
                  'C:\\Users\\james\\Library\\AutoPkg\\RecipeRepos\\com.github.autopkg.homebysix-recipes': {'URL': 'https://github.com/autopkg/homebysix-recipes'},
                  'C:\\Users\\james\\Library\\AutoPkg\\RecipeRepos\\com.github.autopkg.recipes': {'URL': 'https://github.com/autopkg/recipes'},
                  'C:\\Users\\james\\Library\\AutoPkg\\RecipeRepos\\com.github.hjuutilainen.autopkg-virustotalanalyzer': {'URL': 'https://github.com/hjuutilainen/autopkg-virustotalanalyzer'},
                  'C:\\Users\\james\\Library\\AutoPkg\\RecipeRepos\\com.github.jgstew.jgstew-recipes': {'URL': 'https://github.com/jgstew/jgstew-recipes'}},
 'RECIPE_SEARCH_DIRS': ['.',
                        '~/Library/AutoPkg/Recipes',
                        '/Library/AutoPkg/Recipes',
                        'C:\\Users\\james\\Library\\AutoPkg\\RecipeRepos\\com.github.hjuutilainen.autopkg-virustotalanalyzer', 
                        'C:\\Users\\james\\Library\\AutoPkg\\RecipeRepos\\com.github.autopkg.hansen-m-recipes',
                        'C:\\Users\\james\\Library\\AutoPkg\\RecipeRepos\\com.github.autopkg.homebysix-recipes',
                        'C:\\Users\\james\\Library\\AutoPkg\\RecipeRepos\\com.github.jgstew.jgstew-recipes']}

AutoPkg output

python ..\autopkg\Code\autopkg run -v Microsoft/VSCode-Win64.bigfix.recipe.yaml
WARNING: Library 'xattr' unavailable. Defining no-op implementation.
WARNING: Failed 'from Foundation import NSDictionary' in autopkglib.MunkiInstallsItemsCreator
WARNING: Failed 'from Foundation import CFPreferencesCopyAppValue' in autopkglib.MunkiSetDefaultCatalog
WARNING: Failed 'from Foundation import NSPredicate' in autopkglib.StopProcessingIf
Looking for com.github.internal.download.VSCode-Win64...
Did not find com.github.internal.download.VSCode-Win64 in recipe map
Search GitHub AutoPkg repos for a com.github.internal.download.VSCode-Win64 recipe? [y/n]: n       
We are not currently searching on-disk for recipes; if you intended to run a specific file not in the recipe map, please pass in an absolute file path instead.
Could not find parent recipe for Microsoft/VSCode-Win64.bigfix.recipe.yaml
**load_recipe time: 8.398818146000849

Nothing downloaded, packaged or imported.

Expected behavior

To be able to run autopkg recipes in the current working directory that are actively being developed.

Version (please complete the following information):

• OS version: Windows 10
  • AutoPkg Version: dev branch

[jgstew]

I think I was able to solve this by explicitly adding the absolute path to the current working directory to the RECIPE_SEARCH_DIRS and then deleting the map file to get autopkg to recreate it.

I'm not sure this is an ideal solution, but it was less annoying than I was thinking it might be.

[nmcspadden]

Yeah this is currently a known limitation. If you look through my recent commits, I've left comments in the code to that effect - currently, the -d argument is functionally useless.

I need to rethink that part of the process a bit more and I haven't yet concluded on what strategy I want to use, but I don't want to lose the ability to pass in arbitrary directories, as you discovered.

On Fri, Aug 11, 2023 at 4:02 PM JGStew @.***> wrote:

I think I was able to solve this by explicitly adding the absolute path to the current working directory to the RECIPE_SEARCH_DIRS and then deleting the map file to get autopkg to recreate it.

I'm not sure this is an ideal solution, but it was less annoying than I was thinking it might be.

— Reply to this email directly, view it on GitHub https://github.com/autopkg/autopkg/issues/876#issuecomment-1675399054, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJFTX4GAITEZ3XYSE6ND4DXU2MUZANCNFSM6AAAAAA3NMO5AE . You are receiving this because you were assigned.Message ID: @.***>

--

Nick McSpadden @.***

[jgstew]

I'm glad that I found a solution for my specific use case, so I'm all good there, but I do think the ability to pass in arbitrary directories OR the ability to use the current working directory... in the cases where those are not in the recipe search dirs setting, would be ideal.

I think the issue is, it doesn't make sense for those values to end up in the same map file as all the others because they are slightly more ephemeral.

It is almost like you'd have to either fall back on older methods for recipe ID lookup for those arbitrary folders, OR you would have to build the map file in that actual directory every single run, or only if the map file was older than X hours and you would have to search the arbitrary directory map first then fallback to the global map file.

It definitely gets rather complex and figuring out the "correct" solution isn't easy.

I started investigating using the dev branch of autopkg entirely for this map feature because it is so so fast, but it is also helpful to make sure that what I'm doing isn't going to be broken by new changes. I'm now satisfied that I think I'm okay there with a slight change, but it would be great to figure out a solution for arbitrary directories.

[jgstew]

Our weekly test builds of all recipes was taking ~21.5 hours to complete for ~600 recipes.

Since switching to autopkg dev branch and getting the recipe map file working, it now takes ~6 hours, which is a 3.5x speed up.

[nmcspadden]

That is absolutely fantastic to hear!

On Mon, Aug 14, 2023 at 10:02 AM JGStew @.***> wrote:

Our weekly test builds of all recipes was taking ~21.5 hours to complete.

Since switching to autopkg dev branch and getting the recipe map file working, it now takes ~6 hours, which is a 3.5x speed up.

— Reply to this email directly, view it on GitHub https://github.com/autopkg/autopkg/issues/876#issuecomment-1677714846, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJFTXYJ24QGXT24FJJ7S4TXVJKZHANCNFSM6AAAAAA3NMO5AE . You are receiving this because you were assigned.Message ID: @.***>

--

Nick McSpadden @.***

[nmcspadden]

This should now be generally resolved, and AutoPkg should be able to correctly run recipes by path. Please re-test your procedure and see if the existing flows are satisfactory, and re-open this issue if you've got more issues.