git-annex special remote for the open science framework

[adswa]

Guidelines

We are very excited to meet you at the 2020 OHBM Brainhack 🎉 To submit a project, you need to be an attendee of the 2020 OHBM Brainhack. We ask you to register first over here. Thank you!

We have prepared a checklist to help with your project submission. Here is how to proceed:

1. Before filling in any part, please submit this issue
2. Check items in the checklist below as you go through them
3. Once you are done (at least all 'required' items must be provided), please delete the "Guidelines" section add a comment saying 'hi @ohbm/project-monitors: My project is ready!'

Thank you!

After step 1 (issue submitted), we will assign a 'project monitor' to follow your submission. If at any time you need help or anything is unclear, please add a comment and ping your project monitor. Our team is here to help!

---

Project info

Title: git-annex special remote for the open science framework

Project lead: Adina Wagner, @adswa

Timezone: UTC+2

Hub: Europe, Middle East and Africa

Description: The Open Science Framework (OSF) is an amazing infrastructure for open science. In this project, we attempt to create a git-annex special remote implementation to leverage OSF filestorage and make the data storing that OSF provides even more useful. The git-annex OFS special remote would allow to transform OSF storage into git-annex repositories. Files in OSF storage could thus be consumed or exported fast and easily via git-annex or datalad, and published to repository-hosting services (GitHub, GitLab, Bitbucket, ...) as lightweight repositories that constitute an alternative access to the data stored on the OSF - that is: you can git clone a repository from for example GitHub and get the data from the OSF from the command line or in your scripts.

Link to project: https://github.com/adswa/git-annex-remote-osf There is no code yet - we will assemble a group of interested Brainhackers and start from scratch

Mattermost handle: adina

Goals for the OHBM Brainhack

• Familiarize ourselves with git-annex's concept of special remotes
• Explore the OSF API and OSF's waterbutler to interact with OSF's various supported file storage services
• Get started with a Python-based special-remote implementation
• Set up appropriate documentation
• Short term goal: export and import data from public OSF projects from and into git-annex repositories
• Nice-to-have's/Long(er) term development goals: Extend functionality to private repositories (requires token-based authentication)

Good first issues:

• Set up the allcontributors-bot in the project repository to acknowledge contributions
• Create a public and a private project on the Open Science Framework for us to play with. The more comprehensive this task is carried out, the better. Here's a sketch: Add (any) data (honestly, any - cat content, your favorite recipe, a funny video, ...) to different storage providers (OSFStorage, GDrive, Dropbox, ...) to a public and a private repository on the OSF. Make sure that some data exists in a single version, but also take data and update it with a modified version, using the OSF's version control features.

You can find more good first issues in the project repository: https://github.com/adswa/git-annex-remote-osf/issues :hugs:

Skills:

• Technical writing: Good user documentation is essential. Depending on what you are interested to do this could take the form of a README, a readthedocs or Mkdocs-based technical documentation, tutorials, a usecase in the DataLad handbook, or anything else that you can think of.
• Experience with the Open Science Framework: If you are a user of the Open Science Framework, let us know how you use it, walk us through a public project of yours, provide us with projects to use for testing, your ideas and needs, ... If you have used the OSF then you will be a great help!
• Python programming: We aim for a special remote implementation in Python. There are good templates and many examples that we will use and learn from along the way.
• Interest in git-annex or experience with it: We are looking for git-annex experts and novices alike - and everything in between. If you saw "git-annex" and thought "uuhhhh, interesting :thinking:" then come get on board!
• Knowledge about HTTP requests/authentication

Chat channel: https://mattermost.brainhack.org/brainhack/channels/git-annex-osf

Video channel Please have a look at the Mattermost channel (pinned posts) for the URL of the video channel, or alternatively, please contact to @adswa.

Project submission

Submission checklist

Once the issue is submitted, please check items in this list as you add under 'Additional project info'

Please include the following above (all required):

• [x] Link to your project: could be a code repository, a shared document, etc. See here
• [x] Include your Mattermost handle (i.e. your username). If you do not have an account, please sign up here.
• [x] Goals for the OHBM Brainhack: describe what you want to achieve during this brainhack. See here.
• [x] Flesh out at least 2 "good first issues": those are tasks that do not require any prior knowledge about your project, could be defined as issues in a GitHub repository, or in a shared document, cf here.
• [x] Skills: list skills that would be particularly suitable for your project. We ask you to include at least one non-coding skill, cf. here.
• [x] Chat channel: A link to a chat channel that will be used during the OHBM Brainhack. This can be an existing channel or a new one. We recommend using the Brainhack space on mattermost, cf. here.
• [x] Video channel: Please create a video channel that will be used during the OHBM Brainhack and share it in your chat channel above. This can be an existing channel or a new one. For instance a jitsi meet room, cf. here.

You can also include information about (all optional):

• [ ] Number of participants, cf. here
• [ ] Twitter-size summary of your project pitch, cf. here
• [x] Provide an image of your project for the OHBM brainhack website
• [ ] Set up a kanban board on your repository to better divide the work and keep track of things, cf here
• [x] Project snippet for the OHBM Brainhack website, cf. here

We would like to think about how you will credit and onboard new members to your project. We recommend reading references from this section. If you'd like to share your thoughts with future project participants, you can include information about (recommended):

• [ ] Specify how will you acknowledge contributions (e.g. listing members on a contributing page).
• [ ] Provide links to onboarding documents if you have some.

[Remi-Gau]

hey @adswa this looks great and super useful. I added some labels to this issue. make sure you are fine with those or if you want to change or add more. I guessed that a minimal knowledge of git was needed but nothing above git-1 feel free to update.

[Remi-Gau]

Also let us know if there is an image for this project we can use for the website. :-)

[ufangYang]

Hiya @adswa. I will be taking your project for the @ohbm/hack2020-social_media

[adswa]

Thanks for the tags @Remi-Gau :)

Also let us know if there is an image for this project we can use for the website. :-)

I've added a quick logo in https://github.com/adswa/git-annex-remote-osf/blob/master/brainhack/git-annex-osf-logo.png, hope this suffices! :)

[adswa]

Ah, I just read https://github.com/ohbm/hackathon2020/blob/master/.github/ISSUE_TEMPLATE/handbooks/projects.md#project-snippet-for-the-ohbm-brainhack-website -- I'll PR this soon!

[Remi-Gau]

Perfect.

You will give our welcome bot a test run.

[Remi-Gau]

I've added a quick logo in https://github.com/adswa/git-annex-remote-osf/blob/master/brainhack/git-annex-osf-logo.png, hope this suffices! :)

This logo is all asked for and so much more.

[adswa]

the bot is f-ing fantastic! :upside_down_face:

[adswa]

Hiya @adswa. I will be taking your project for the @ohbm/hack2020-social_media

thank you @ufangYang!

[lnnrtwttkhn]

Hi @adswa, I would be happy to join this project during the hackathon.

Actually, I wondered before if DataLad could also be used with Seafile as the special remote, so maybe while figuring out how it could work with OSF, I would also learn how to use it with Seafile.

[adswa]

Hi @lnnrtwttkhn, great to have you on-board! :-)

Actually, I wondered before if DataLad could also be used with Seafile as the special remote, so maybe while figuring out how it could work with OSF, I would also learn how to use it with Seafile.

Yes, it should work. I have never done it before myself (I didn't know about Seafile until now), but it works via the rclone special remote. There is a walk through on how to set everything up via rclone for Dropbox in the handbook (http://handbook.datalad.org/en/latest/basics/101-138-sharethirdparty.html). I suppose it should be possible to do it in a highly similar fashion for seafile, and if you end up doing it, it would actually be cool to have a write-up of it (handbook contribution? :heart_eyes:). Also, as another pointer, we've started work around a wrapper for rclone, because it supports so many third party services, and I think its in a workable state ( https://github.com/datalad/datalad/pull/4162 and https://github.com/datalad/git-remote-rclone). The wrapper should at some point replace the workflow thats currently described in the handbook, so if you're up to giving those a test-run, that would be superb!

[lnnrtwttkhn]

Hi @adswa, that sounds great! I will check out the links and see if I can get it to work!

[jhlegarreta]

@adswa do you have a video channel for your project? Would you add that information to the mattermost channel, please? Thanks.

[adswa]

Yes, we took it out to prevent "zoom-bombing". It is now in the header and pinned message of the mattermost channel.

[sappelhoff]

Unfortunately I won't have time to participate in this Brainhack, but I think this project is really exciting, and I'll try to follow it as much as time permits.

I just wanted to share how I used Datalad and OSF in conjunction in the past:

1. using https://github.com/templateflow/datalad-osf ... however, the repo seems to be currently unmaintained
2. using this "recipe" I wrote in the form of a blog post

This may or not be helpful, but I thought it doesn't hurt to share :-)

[adswa]

thanks for sharing, @sappelhoff!

[kousu]

Hi @adswa, I'm working on managing these issues with @neuropoly, and this seems like a fascinating challenge, and also pretty accessible. Is it too late to get in on it?

[adswa]

Not at all, we haven't even pitched. We'll be pitching in 1.5 hours in the EMEA hub. :)