search

enram / data-repository

Data quality assessment
https://enram.github.io/data-repository/
MIT License
3 stars 1 forks source link

new profiles are not being uploaded #47

Closed adokter closed 6 years ago

adokter commented 6 years ago

@stijnvanhoey I just checked the online repo, and none of the newly generated data has been made available for some time. I checked the baltrad ftp server, but the vertical profiles are still being generated there. So I think something may be wrong on your side of the pipeline.

Strange thing is that some of the files have been updated recently, e.g. the zipfiles here: http://enram.github.io/data-repository/?prefix=nl/hrw/2017/

Could it be you are writing 2018 files to 2017 folder?

stijnvanhoey commented 6 years ago

I recently updated all zip files during the repo cleaning this week, so that additional update is correct. The pipeline was not giving any errors recently, but I'll give it a full check on monday; I'm currently without PC till monday. Not so easy on smartphone on bike ;-)

stijnvanhoey commented 6 years ago

@adokter, after a first check on the logs, the transfer worked each day, but just couldn't get any data from the Baltrad FTP. E.g. 2018-01-01, 2018-01-02 and 2018-01-03, the transfer procedure reports no new available data at the Baltrad side.

Just as check: the procedure uses the (original) volbird@odc.baltrad.eu instead of co-0181.c.ipeer.se, as the original alias should still work. Could that be the reason?

And another check, we decided previously to only transfer the _vp_ files and not the pvol. Is that naming still ok and active?

adokter commented 6 years ago

Hi Stijn, Yes I see now too that some of the profiles are simply missing from the baltrad side, we will look into it. Closing the issue here.

adokter commented 6 years ago

@stijnvanhoey @peterdesmet Since a few days the data seem to flowing in again, maybe something broke in Sweden which was solved when people got back from Christmas.

Without any doubt such hickups will happen more often in the future. It would be good if we could see relatively quickly when certain radars stop providing data. A crude but simple solution would be if next to each country folder and radar folder the date of latest addition would be shown at http://enram.github.io/data-repository/. Is that a tweak that could be made before the start of this spring migration season (Mar 1), or does that require a real project?

peterdesmet commented 6 years ago

@adokter I've created a new issue with this feature request: https://github.com/enram/data-repository/issues/48