search

MAAP-Project / maap-hec-aws

2 stars 0 forks source link

Data Manifest Capability (In) #85

Closed wildintellect closed 1 year ago

wildintellect commented 2 years ago

Description

For jobs, the data required ideally should be clearly specified upfront. This is equivalent to the existing DPS input yaml Download=True. HEC jobs will need a similar way to supply urls (https or s3), likely in CWL format to the stage-in executor.

Long term, future iterations (another ticket) may modify this to be a json in STAC format that is translated or iterated by the CWL stage-in. This is not part of this ticket.

Acceptance Criteria

wildintellect commented 1 year ago

In discussions with @jplzhan and @jjacob7734 this is going to be an array of values passed to the submitJob. @jjacob7734 please confirm the name of the variable you want to use.

Example

file_inputs: ["s3://URI1", "https://URI2"]
jjacob7734 commented 1 year ago

I think the name of the variable can be anything. In the CI/CD @jplzhan will mark that input variable as a stage_in variable in the application descriptor JSON file. The ADES will use that marking to identify the stage in variable, whatever it is named.

Max, can you confirm that the science users building the application can use any keyword they like in the manifest?

moronidav commented 1 year ago

@jplzhan confirmed today that whis was resolved in verified. @wildintellect Feel free to close this out.