openSNP / snpr

The sources of the openSNP website
http://opensnp.org
MIT License
174 stars 46 forks source link

Create new json endpoint #513

Open arvkevi opened 6 years ago

arvkevi commented 6 years ago

A list of URLs for user's genotype files would be helpful for Open Humans integration.

philippbayer commented 6 years ago

That sounds good and shouldn't be too hard to implement - something living at https://opensnp.org/genotypes.json and just printing all genotypes with date uploaded, associated user ID, genotype ID, and URL link!

philippbayer commented 6 years ago

If it has to happen now, the users JSON already includes all genotypings too: http://opensnp.org/users.json You'd just have to iterate over each user, via https://github.com/openSNP/snpr/wiki/JSON-API

gedankenstuecke commented 6 years ago

We already have this at the https://opensnp.org/users.json endpoint. The problem is that we have too many users by now! As it's not paginated and just returns all users it's too big to be useful:

  1. the JSON blob is large and takes long to process
  2. it takes us by far too long to produce the JSON before delivering it

What @arvkevi and I had discussed was an endpoint that lives at https://opensnp.org/users/1.json and returns for the user_id=1 all files we have.

This would also play nicely with the Open Humans integration we already have in place, as we upload a custom JSON to Open Humans that contains a link to the openSNP user-profile. So it would be easy for someone to grab the JSON for that user instead of having to parse our profile HTML.

gedankenstuecke commented 6 years ago

I should probably give some more context: @arvkevi and I discussed this earlier today (yesterday for @philippbayer :joy:) as he's building http://openimpute.com/. That project takes genotyping files that are stored in Open Humans, imputes them and then uploads an imputed version back to Open Humans.

So far he can support VCF files from other Open Humans projects as he can get the download-URLs easily from Open Humans. But the openSNP integration is special as we don't deposit any actual data into a person's Open Humans account, but rather just a file that links to the openSNP user account.

arvkevi commented 6 years ago

Well summarized, thanks for considering the request 😄