Feature: make CI action to register new curation files to CodeOcean

[alejoe91]

User story

As a user, I want the curated files registered to Code Ocean, so I can analyze them on that platform.

Acceptance criteria

• [ ] When a curation file is saved to this directory, then a ci action uploads the folder to S3
• [ ] After the folder is uploaded to S3, another CI action will register the data asset to Code Ocean with
  • string tags ["ecephys", subject_id, "curated"]
  • custom metadata tags { "modality": "extracellular electrophysiology", "data_level": "derived data", "subject id": subject_id }
• [ ] The folder should obey DerivedDataDescription naming conventions. For example: ecephys_123456_2023-01-02_12-12-12_curated_2023-01-03_12-12-12

Sprint Ready Checklist

• [ ] 1. Acceptance criteria defined
• [ ] 2. Team understands acceptance criteria
• [ ] 3. Team has defined solution / steps to satisfy acceptance criteria
• [ ] 4. Acceptance criteria is verifiable / testable
• [ ] 5. External / 3rd Party dependencies identified
• [ ] 6. Ticket is prioritized and sized

Notes

Add any helpful notes here.

[jtyoung84]

@alejoe91 Can you review/modify the acceptance criteria to get it how you want it? Alternatively, I can also write a CI script that will save the json file to a database.

[alejoe91]

@jtyoung84 I think @dyf wrote it up quite nicely! That's exacly what I had in mind :)

[jtyoung84]

@alejoe91 Do you know if it's possible to easily save these directly to a bucket in s3?

[alejoe91]

It's probably doable if we set AWS credentials as secrets. Would that be easier then to register them in CO?

[jtyoung84]

I can set up a lambda function to watch for new files that land in an s3 bucket. The lambda function can then perform the registration in Code Ocean. It's mostly that I feel like we can remove the "saving to github" step if we're just copying the data over to s3 anyways. I don't like the idea of using GitHub as a database if we can avoid it.

[alejoe91]

Yeah, but that's currently how FigURL does it, because it uses github login so it can actually write to a repo...so I don't think we can remove the Save to Github for now