programmatic way to download datasets from OSF

[jedyeo]

use OSF api to do so

[ninamiolane]

yesss 💯

[ninamiolane]

I assume this has been addressed by PR #25 . Closing. please reopen if that is not the case.

[geoffwoollard]

@jedyeo and @ninamiolane I don't see any functionality for downloading in https://github.com/compSPI/ioSPI/pull/25, so I've reopened.

[geoffwoollard]

Re Arjun's question on Slack

Arjun Swani 23 Feb 2022 at 7:57 PM Hi all, Had a question regarding the workflow for uploading datasets generated using TEM wrapper - Currently, we have a simulator agnostic library "datasets.py" in ioSPI that can be used for generic read/write/upload functions related to OSF.io. for eg: create a node, get list of all nodes etc. I am writing a function using these generic methods in ioSPI to upload datasets generated by temWrapper. This function takes the meta data files and particle slices generated specifically by the TEM wrapper to label, tag and upload datasets. Considering that this is TEM specific functionality (worth noting that it does not import anything from simSPI), where would be the best place to keep it? Would this be another method in datasets.py in ioSPI or would it live in simSPI? A rough draft of the functions can be found here : https://github.com/compSPI/simSPI/blob/temUpload/simSPI/temUpload.py

@arjunsingh3600 , I think that we can just have a bare bones method for downloading (and uploading) to ioSPI.

As a developer of ioSPI, you can make a convention as what input the uploader takes, and what output the downloader gives. Config files, dictionaries, etc. In PRs, this can be discussed at a more granular detail. And it can be revisited and refactored as times goes on and we upload more things to OSF and see how it's working.

To interface with how ioSPI uploads and downloads, I think you can have simulator specific parsing things in simSPI.

• For instance you can parse output from tem_simulator in simSPI and get it into a dictionary/config file that is a certain format needed by your ioSPI uploader. So the simulator specific things stay in simSPI.
• And you can download from OSF with ioSPI make it output a certain format. Then you can parse this further if needed in simSPI to get things ready for actually doing another simulation (e.g. load config files, call run on a specific simulator). If the metadata needed to reproducibly run on OSF is needed to be parsed to make it ready actually do the simulation on simSPI, I think this should be in simSPI

It doesn't seem that complicated to me

1. simSPI does all the simulating from a convention of input config files and outputs a convention of metadata and simulated stuff (latents and simulated-observes)
2. ioSPI uploads files, whatever they are, so OSF.

What ioSPI should have is OSF specific code. simSPI should be agnostic to OSF.

@ninamiolane , what do you think?

[fredericpoitevin]

@arjunsingh3600 and team, where do we stand on this following yesterday's discussion?

[jedyeo]

Programmatic way to download datasets can be found in the tutorial notebooks. See https://github.com/compSPI/simSPI/pull/87/

[ninamiolane]

Yes, but:

• users of ioSPI might not get the idea of going to see in simSPI to learn this.
• ioSPI is supposed to be simSPI agnostic Functionalities to download from OSF are io-utils, that have their place as functions from the ioSPI package, not as lines of code in a notebook somewhere in simSPI.

[geoffwoollard]

Also @jedyeo , in the notebook https://github.com/compSPI/simSPI/blob/master/notebooks/tem_tutorial.ipynb I do not see functionality to download datasets from OSF

I don't even see functionality to download data from OSF in https://github.com/compSPI/ioSPI/blob/master/ioSPI/datasets.py @arjunsingh3600 weren't you working on this?

[ninamiolane]

Any update on this?

[geoffwoollard]

I'll ask @arjunsingh3600

[ninamiolane]

Thanks! I will do it though, because we need it urgently for next week with the hackthon :)

[ninamiolane]

Closed by #70