search

AllenNeuralDynamics / aind-data-transfer-service

FastAPI service to run data compression and transfer jobs on the hpc
MIT License
1 stars 0 forks source link

mount field to process new data assets in a pipeline #111

Closed arielleleon closed 3 months ago

arielleleon commented 3 months ago

Is your feature request related to a problem? Please describe. I cannot specify a mount point for a capsule in the current architecture. This is required to trigger a pipeline with a new data asset.

Describe the solution you'd like I would like a mount field to the BasicJobUploadConfig so that transfer-service can specify a mount point when a pipeline is processed.

dyf commented 3 months ago

@arielleleon @jtyoung84 I'm wondering if we want also to just have a library of configurations that know how to set mount points and other properties by default. For example the ecephys cares about two mount points (one for the data, one for the noise classifier model). The single-plane-ophys and multiplane-ophys pipelines likewise would have their own.

Concern with this is that I think the trigger capsule needs to know about this as well.

arielleleon commented 3 months ago

@dyf @jtyoung84 I'm fine with a default configuration for platform / modality. But do we need to care about the trigger capsule anymore? Looks like the aind-data-transfer basic-job can take in any capsule id

dyf commented 3 months ago

@arielleleon If a scientist wants to trigger reprocessing without re-uploading themselves, one way to do that is to use the trigger capsule app panel. It is (debatably) convenient to be able to do this within Code Ocean rather than learning to use the API.

jtyoung84 commented 3 months ago

@arielleleon Currently, the trigger capsule is run as default. It registers a data, runs a pipeline, monitors the pipeline, and captures the results once the pipeline is finished. We can add update the BasicUploadJobConfigs class to handle other cases, although I'd like to keep things backwards compatible as much as possible.

@dyf I discussed this Alessio. I think it should be fine having defaults in the trigger capsule repo itself. I'm probably fine with having the defaults checked in the repo, but we can pull them from a database if needed.