mjaquiery
commented
3 months ago This is going to be very challenging to implement with our current architecture. The way we have things set up, Harvesters upload files in a two-stage process that involves editing the data so that they can map columns appropriately for meta-analysis. This makes it difficult to replicate the Harvester's behaviour on the frontend or backend without duplicating the Harvester code.
Some approaches we might try for implementing this functionality:
Let the Harvester code do the work
We could allow users to upload files to temporary directories on the backend server, and run a Harvester::harvest process on those files. This would require spoofing quite a lot of stuff on both Harvester and Backend side, though, and would likely break frequently when trying to update either service.
Let the Frontend do the work
We could hold uploaded files in memory and require the user to manipulate them within the Frontend until they are satisfactory data files for backend storage. We could then allow them to be uploaded and adjust the Backend code to allow uploaded files to come from non-Harvester sources (or create a dummy Harvester that represents user-uploaded files).
This requires a lot of the Harvester code to be reimplemented in React/TypeScript.
It would be great if we had functionality to upload data via the frontend. Ideally, this could enable sharing data between collaborators or individual users without needing harvesters running locally. I think this would be most beneficial when collaborators wanted to share few data files, otherwise it would make more sense to have a harvester running locally. This could also enable sharing of simulated data in addition to experimental data.
Original issue by @BradyPlanden https://github.com/Battery-Intelligence-Lab/galv/issues/66#issue-1655762994