Revise VM harvester script to ensure it resumes after daily server backup

[helendallas]

At the moment the VM harvester stops because each 24hours at 00:00.

Please improve the script to enable it to resume when it stops.

[dimasciput]

Checked on June 10th 2024, the api seems to be down.

[helendallas]

From Rene:

The API is available again. The base URL of the API calls needs to be changed from ​api​.adu​.org​.za to api.birdmap.africa. For example: http://​api​.adu​.org​.za​/vmus​/v2​/project​/OdonataMAP becomes https://api.birdmap.africa/vmus/v2/project/OdonataMAP

VM API documentation:

https://api.birdmap.africa/vmus/v2/ND/docs/#File:index.php

[dimasciput]

thanks @helendallas , will work on it today.

[dimasciput]

Hi @helendallas

I have made the changes, and there are a lot of things that will be updated. So, I'm going to work on this in the multi-tenant environment only. When we migrate the production of FBIS to the multi-tenant server, this process should automatically run every month. It will check daily if the last process has stopped; if it has, it will simply continue from where it left off.

[helendallas]

Thanks @dimasciput . Sounds good. I can't check as not activated to harvest, so records still show 2023. I will close ticket and have to create new one if there are issues in the multi-tenant production once migrated.

[helendallas]

@dimasciput I cant test this. Please can you start harvesting FrogMap and OdonataMap. Then I will check back in a day and see if the new records have been brought down. Thanks

[helendallas]

@dimasciput Did you start a harvest for these data?

[helendallas]

Sorry @dimasciput The data is still only till early 2023. Not working. Please check again

https://freshwaterbiodiversity.org/map/#site-detail/taxon=&search=&siteId=&collector=&category=&yearFrom=&yearTo=&months=&boundary=&userBoundary=&referenceCategory=&spatialFilter=&reference=&endemic=&invasions=&conservationStatus=[]&modules=36&validated=&sourceCollection=[%22virtual_museum%22]&module=occurrence&ecologicalCategory=&rank=&siteIdOpen=&orderBy=name&polygon=&dst=&ecosystemType=River,Wetland,Open%20waterbody,Unspecified

https://fbis.sta.do.kartoza.com/map/#site-detail/taxon=&search=&siteId=&collector=&category=&yearFrom=&yearTo=&months=&boundary=&userBoundary=&referenceCategory=&spatialFilter=&reference=&endemic=&invasions=&conservationStatus=[]&modules=36&validated=&sourceCollection=[%22virtual_museum%22]&module=occurrence&ecologicalCategory=&rank=&siteIdOpen=&orderBy=name&polygon=&dst=&ecosystemType=River,Wetland,Open%20waterbody,Unspecified

https://freshwaterbiodiversity.org/map/#site-detail/taxon=&search=&siteId=&collector=&category=&yearFrom=&yearTo=&months=&boundary=&userBoundary=&referenceCategory=&spatialFilter=&reference=&endemic=&invasions=&conservationStatus=[]&modules=21&validated=&sourceCollection=[%22virtual_museum%22]&module=occurrence&ecologicalCategory=&rank=&siteIdOpen=&orderBy=name&polygon=&dst=&ecosystemType=River,Wetland,Open%20waterbody,Unspecified

https://fbis.sta.do.kartoza.com/map/#site-detail/taxon=&search=&siteId=&collector=&category=&yearFrom=&yearTo=&months=&boundary=&userBoundary=&referenceCategory=&spatialFilter=&reference=&endemic=&invasions=&conservationStatus=[]&modules=21&validated=&sourceCollection=[%22virtual_museum%22]&module=occurrence&ecologicalCategory=&rank=&siteIdOpen=&orderBy=name&polygon=&dst=&ecosystemType=River,Wetland,Open%20waterbody,Unspecified

[dimasciput]

@helendallas this is the anurans

https://freshwaterbiodiversity.org/map/#site-detail/taxon=&search=&siteId=&collector=&category=&yearFrom=&yearTo=&months=&boundary=&userBoundary=&referenceCategory=&spatialFilter=&reference=&endemic=&invasions=&conservationStatus=[]&modules=36&validated=&sourceCollection=[%22virtual_museum%22]&module=occurrence&ecologicalCategory=&rank=&siteIdOpen=&orderBy=name&polygon=&dst=&ecosystemType=River,Wetland,Open%20waterbody,Unspecified

[helendallas]

@dimasciput Frogs include 2024.

Still need to fix Source References

• 

Still not done for Odonates - 2023 is last record.