RFC: Job Dependencies on Products

[amzn-mike]

Summary:

Job dependencies will be expanded to allow Builders to provide a list of products they specifically depend on. Asset Processor will also record the hash of every product as it's created. This will allow Asset Processor to only queue the dependency when the relevant products have actually changed.

What is the relevance of this feature?

Without this, long dependency chains can cause large numbers of files to run through CreateJobs, taking considerable time. This will reduce the number of CreateJobs calls to the ones which actually need to do work. All of the builders involved in the process of compiling shaders/materials are the main use case, but any builder with Job Dependencies can make use of the feature.

Feature design description:

AssetBuilders will be able to include a list of SubIds when declaring a Job Dependency. Asset Processor will store this list in the Asset Database. Whenever a file changes, Asset Processor will look up the list of direct dependencies and will only immediately queue those which do not include any SubIds. After a ProcessJob call is finished and we've collected the output products, AP will hash each product and compare the hashes with those stored in the database to get a list of updated products. AP will again look up the list of direct dependencies and this time will queue up any which list SubIds that are included in the list of updated products.

As a brief overview of the existing types of dependencies, AP currently supports 2 types of build-time dependencies: Source Dependencies and Job Dependencies, as well as 1 type of run-time dependency: Product Dependencies.

In the diagrams below, "No regular dependencies found" refers to Source Dependencies or Job Dependencies which do not make use of this feature (meaning no SubId was specified). "Find dependents on me" means the AP is searching for all files which depend on the source file, in this case, StandardPBR.materialtype

Example flows (not representative of current behavior):

Before:

After:

After, meaningful change:

Technical design description:

AssetBuilderSDK::JobDependency will be updated to have a new field: AZStd::vector<AZ::u32> m_productSubIds. This list will be stored in the asset database SourceDependencies table as a comma-separated list SubIds, as this avoids the need for a new table and it shouldn't be necessary to write queries against individual SubId values, we just need to store/retrieve the value in bulk. Both the serialization and behavior context reflection will need to be updated. This will be available to all builders, including python based builders.

The Products table will also be updated to have a new Hash column. The hash of the product will be computed during AssetProcessed_Impl when preparing to store the products to the database.

GetSourceFilesWhichDependOnSourceFile will be updated to take a list of SubIds. If the list is empty, it will only return entries from the SourceDependencies table which have an empty SubIds column. This will result in the original call to this function (during CheckSource) returning only dependencies which do not care about a specific product. By default this will result in the same behavior as before. Builders which are not updated to provide the new info will return an empty list and still run in the exact same manner as before.

In AssetProcessed_Impl, after analyzing the list of products a job just produced, the list of prior products will be subtracted from the list of new products to produce a list of UpdatedProducts (the file hash will be the determining factor of difference). The resulting list of UpdatedProducts will be provided to GetSourceFilesWhichDependOnSourceFile which will return any direct dependencies with a non-empty list of SubIds where at least 1 of the SubIds is in the list of UpdatedProducts. All of these dependencies will then be put in the processing queue.

What are the advantages of the feature?

• Trivial changes that result in the same product output ideally results in no dependencies being queued up.
• When only some of the outputs have changed, only a subset of the dependencies may need to be reprocessed.
• Implementation is fairly simple.
• Usage by builders is optional and can be as simple as a 1-line addition.

What are the disadvantages of the feature?

There shouldn't be any direct disadvantages to using the feature.

How will this be implemented or integrated into the O3DE environment?

• The Asset Database queries will need to be updated to add the Hash and SubIds columns.
• AssetBuilderSDK needs to be updated to add the new field for storing the SubId list.
• AssetBuilderManager needs to be updated to hash the products, generate the list of updated products, find relevant dependencies and queue them up.

Are there any alternatives to this feature?

The originally proposed solution to this problem was to allow the creation of "intermediate" jobs in a way that would allow jobs to create other jobs in a chain. Intermediate jobs is a more powerful feature which can also be used here but it is a more complex feature. This proposal allows a much simpler solution which allows builders a 1-line change for cases where all that is needed is a dependency on a specific product. Additionally, this does not prevent the inclusion of such a feature in the future.

How will users learn this feature?

• Asset Builder documentation will need to be updated to explain this feature.
• Usage is as simple as calling m_productSubIds.push_back(subId) on a JobDependency in the builder's CreateJobs function

Are there any open questions?

• Do we need to support wildcards? There may be a use case where a builder does not know all of the products it depends on. The downside to supporting this is typically a much more complicated implementation with more significant performance penalties.

[lemonade-dm]

This RFC looks good to me. It's opt-in, so existing builders don't need to change.

The only areas I would look to update is the diagrams showing the source file change workflow. The "Find Dependencies" decision symbol is a bit confusing, as it is not finding the dependencies of the StandardPBR.materialtype, but instead finding what depends on it.

Could you add a note to better explain this?

[amzn-mike]

@lumberyard-employee-dm I've updated the diagrams and added a note as well to make it more clear :)

[lemonade-dm]

Looks good to me. I am signing off on it as representative of SIG-core

[lemonade-dm]

@amzn-mike I am closing out the RFC issue as it has been approved