search

dthonon / Client_API_VN

Export from VisioNature (Biolovision) web site to local postgresql database
GNU General Public License v3.0
3 stars 1 forks source link

Full download with many sources #131

Closed dthonon closed 2 years ago

dthonon commented 3 years ago

In GitLab by @fred.lpo07 on Aug 6, 2020, 15:35

With a configuration containing many sites (like in Auvergne-Rhône-Alpes with 9 sites), full download only download the 2 first sites, not all of them.

To bypass, we need to download by pair of sites and disabling the others.

dthonon commented 3 years ago

In GitLab by @fred.lpo07 on Aug 9, 2020, 22:59

For example, while trying to download observations from Savoie, Haute-Savoie and Auvergne (ordered in that way in config file) , app downloaded perfectly data from Savoie, Haute-Savoie but none from Auvergne and end up with those lines without quit app

...
2020-08-09 15:01:13,769 - INFO - transfer_vn.download_vn - vn74 => Iter: 53, 0 obs, taxo_group: 31, date: 16/07/1907, interval: 1559
2020-08-09 15:01:14,040 - INFO - transfer_vn.download_vn - vn74 => Iter: 54, 0 obs, taxo_group: 31, date: 10/03/1903, interval: 1589
2020-08-09 15:01:14,298 - INFO - transfer_vn.download_vn - vn74 => Iter: 55, 0 obs, taxo_group: 31, date: 02/10/1898, interval: 1619
2020-08-09 15:01:14,299 - INFO - transfer_vn - vn74 => Ending download using controler observations
dthonon commented 3 years ago

Application not ending is issue #125. I will look into the problem of more than 2 sources.

dthonon commented 3 years ago

In GitLab by @fred.perso on Aug 10, 2020, 12:27

It seems to be related to the scheduler, I tried to launch a complete download of the data of Auvergne site, by disabling all other sites and the latest informations are :

2020-08-09 21:56:49,752 - INFO - transfer_vn - Skipping site vn42
2020-08-09 21:56:49,752 - INFO - transfer_vn - Skipping site vn69
2020-08-09 21:56:49,753 - INFO - transfer_vn - Skipping site vn73
2020-08-09 21:56:49,758 - INFO - transfer_vn - Skipping site vn74
2020-08-09 21:56:49,763 - INFO - transfer_vn - Scheduling work for site vnauv
dthonon commented 3 years ago

I tried with 4 sites (vn01, vn07, vn26, vn38) and it works. I will try with Auvergne.

dthonon commented 3 years ago

I downloaded from all sites (observations for 8 days only) with no issue. It could be an issue with Auvergne, as it is a very large site. What are the tuning parameters in YAML configuration ? Could you try with a small interval:

# Optional start and end dates
start_date: 2020-08-01
end_date: 2020-08-10
dthonon commented 3 years ago

Does this still happen, or can I close the issue ?

dthonon commented 3 years ago

In GitLab by @fred.lpo07 on Oct 29, 2020, 11:24

Not retried since issue opening. Let's close it.