Added support for parallelism.

[adahms]

Major re-write of the main build process:

• The build process now enumerates the target files before the build, weeding out duplicates and sorting the list in accordance with alphabetical order.
• The build process now submits each build action to a thread pool executor sized in accordance with the specs of the machine so that multiple builds can take place at the same time, speeding up the overall build process.
• There is now a new 'duplicates' action that allows users to review their pantheon2.yml file for duplicate entries.
• Removed the old 'build_subset' action; all build actions now use the same, global process.

[adahms]

Hey @Levi-Le - would you be available to review this?

Some fairly major changes to clean up the code a bit and do things properly.

[Levi-Leah]

@adahms - sure thing, I'll look at both pull requests as soon as I can

[Levi-Leah]

@adahms, I focused mostly on the pcutil.py file for now. I thought that instead of running similar code twice we can run it just once and capture both return values that we need for pcmd.py.

I also noticed that the script searches directories from top down, and prefers the first pantheo2.yml file it finds, which can potentially be a problem, if someone has multiple repos.

I've added my suggestions, please let me know what you think. Other than that, I find your approach capturing files to be very interesting :)

Also, I think it'll beneficial to also validate the existence of files in the pantheon2.yml file, as I know some of them may have a typo in the path. I'll be submitting a PR for that shortly

[Levi-Leah]

@adahms would you be able to take a look at the last few commits?

1ad0feed17022c78d6112ea7a8600a0782f9505e - modifies the logic in the get_files function. I ran it by Jaromir and he suggested we go this rout, as we don't always have to default to glob.glob

I've also added a function to check if the files in pv2.yml exist in path, which builds on the previous modification on how we use glob.glob f82fb42bf3ba4194e242d1e0ceb039e44367cbbd - core functionality 3904056db688ca041bdfd37425a58c4eaa620ab8 - pcmd.py modification to accommodate for the new functionality

[Levi-Leah]

@adahms, as promissed, here is the breakdown of the latest updates:

b1268d0

• all the code for syntax + yaml validation is now in pcyamlcheck.py
• exists, validate, validate-yaml commands now merged under a single validate command
• the logic of the validation is now as follows:
  1. check syntax. If errors => exit
  2. check keys and if they exist in path (attribute file/images dir). If errors => exit
  3. check if modules/assemblies exist in path
  4. validate modules/assemblies

[adahms]

@Levi-Le - awesome, thanks for the updates!

I adjusted the logic on the get_files function to ensure it's working across a number of repos, and ran through the validate, compile, and duplicates actions for each of the repos in PV2 today.

Everything's coming up good. :)

[adahms]

Merging into main.