search

OGGM / oggm

Open Global Glacier Model
http://oggm.org
BSD 3-Clause "New" or "Revised" License
218 stars 105 forks source link

Find a sensible way to work with open but user-bound data #712

Open fmaussion opened 5 years ago

fmaussion commented 5 years ago

In OGGM we use a couple of datasets which are free to use but hidden behind a personal license agreement and an authentication wall. The best example is TanDEM-X, but other DEMs might apply as well (this needs to be checked).

I would very much provide these data to our users.

@TimoRoth , do you think there is a way to set-up OGGM to use secured ftp and allow our users to provide e.g. username and password via a config file? This would be awesome!

nchampollion commented 5 years ago

But is there a way to put this config file on github (with the data in Zenodo for example or in a ftp) ?

fmaussion commented 5 years ago

But is there a way to put this config file on github (with the data in Zenodo for example or in a ftp) ?

Nono, the point is following: TanDEM-X is freely available but you need to have a user account to access their data. (unlike RGI, where you can simply download the data). They do this because they want everyone to agree to their data policy.

So what we need is that OGGM can access this data (like it does for RGI) but each user need to provide his/her account information by writing it in a file which stays on the user's computer (i.e. OGGM doesn't want to know your password).

fmaussion commented 5 years ago

example of their license: https://geoservice.dlr.de/resources/licenses/tdm90/License_for_the_Utilization_of_90m_DEM_for_Scientific_Use.pdf

TimoRoth commented 5 years ago

FTP is horribly old and insecure, but for their https based download that's easily possible as well.

fmaussion commented 5 years ago

for their https based download that's easily possible as well.

Great! The part that needs to be adapted is here:

https://github.com/OGGM/oggm/blob/64245a45ed2469616fbb81b655eece47737a5455/oggm/utils/_downloads.py#L515

Currently relies on user having ssh access to cluster :roll_eyes:

nchampollion commented 5 years ago

Ok, I think I understand. So basically you build a function in OGGM that looks into your computer to search for username/password (after you had registered into TanDEM-X for example) and download the data through https ... ?

fmaussion commented 5 years ago

@TimoRoth do you have time to work on this or should we do it?

TimoRoth commented 5 years ago

I'm looking into it right now. Implementing it just for TanDEM-X is very straight forward. Just leaves the question where the credentials should be stored. Adding them to the parameters seems risky and might lead to people uploading them somewhere. So I'd probably go for reading them from a special file somewhere in the users homedir.

fmaussion commented 5 years ago

reading them from a special file somewhere in the users homedir.

completely agree!