search

ssl-hep / ServiceX

ServiceX - a data delivery service pilot for IRIS-HEP DOMA
BSD 3-Clause "New" or "Revised" License
19 stars 21 forks source link

Multiple Code Generators for Single ServiceX Deployment #381

Open BenGalewsky opened 2 years ago

BenGalewsky commented 2 years ago

Story

As a ServiceX user I want one ServiceX instance to support multiple code generators and transformers so I can perform all of my work in one instance

Assumptions

  1. Some property in the request would be able to direct the request to a specific code generator
  2. The code generator can suggest which transformer image would pair well with it

Acceptance Criteria

  1. How will we know when this story is complete
BenGalewsky commented 2 years ago

From @gordonwatts

Two possible stories for multi-step:

  1. Someone has uploaded to OpenData a ZIP file that contains lots of different actual input files. So first step would extract files needed from zip file and store them locally (minion? xcache), second step would run the transformer on those files
  2. Output of the uproot transformer is parquet - but user needs a ROOT file. Or output of the C++ transformer is a ROOT file, but user needs parquet. Some sort of second step makes the proper conversion.

Other arguments for this:

  1. A DID finder can fetch different types of data. For example, the CERN OpenData DID finder can access root files for uproot, or CMS Run 1 AOD files for the aod transformer.