FrogMAP - create API to harvest data from virtual museum

[helendallas]

Once the Frog Module has been created and the master list uploaded, then we need to create the api to regularly harvest from occurrence records from FrogMAP on the Virtual Museum.

FrogMAP data can be retrieved from the VM in the same way as the Odonata data, just substitute the name of the project in the API call, the rest should be the same. But unlike the Odonata, FrogMAP gets about 120 new records per month, so the frequency of updating the Kartoza data should be adjusted accordingly.

[helendallas]

@dimasciput The linkages between VM and FBIS columns are similar to OdonataMAP.

Here is the file showing linkages for VM FrogMap and FBIS columns

[jeremyshelton]

Column Header linkages between VM and FBIS (Frogs).xlsx

[dimasciput]

@jeremyshelton @helendallas currently fetching all frog occurrences from virtual museum in testing site.

I think it will be finished tomorrow.

[dimasciput]

@jeremyshelton I have fetched the frogmap data in the testing, but I forgot to add a boundary check so you will see sites outside SA. I have added the check in the latest commit, should I re-harvest the data in testing? Or just harvest it in the live site?

[jeremyshelton]

Thanks @dimasciput Looking good. I am downloading the csv and will have a close look at the data tomorrow. One thing I noticed is the boundary needs to be clipped as per GBIF:

[jeremyshelton]

Hi @dimasciput 👆Responded yesterday, but forgot to hit send 🤦 Happy for it to go straight to live with the boundary check added. But I just want to first check out the data .csv to make sore all is in order before we deploy on live. I requested it 10 hrs ago, but nothing in my inbox yet. Can you perhaps send me a .csv with all the FrogMap data, or must I request another csv download? Thanks

[jeremyshelton]

@dimasciput from the dashboard, I can see that the endemism data from the frog master list are not showing:

[dimasciput]

@jeremyshelton yes please try downloading it again, but maybe try smaller datasets first.

[jeremyshelton]

@dimasciput OK, but spatial and temporal filters are not working on the testing site?

[dimasciput]

@jeremyshelton it's working, I just tested

[dimasciput]

@jeremyshelton please hold the testing, I'm going to deploy the decision support tool feature.

[jeremyshelton]

OK @dimasciput Let me know when it's ready for me to test. Also, I did a hard refresh & can now see the filters 👍

[dimasciput]

@jeremyshelton you can test it now

[jeremyshelton]

@dimasciput I have done a few frog VM data download requests from testing site, but have not yet received a .csv to test?

[jeremyshelton]

@dimasciput FrogMap VM data looking good on testing site.

Just two small issues in the CSV that need fixing: (1) Endemism column is blank but should not be because all of these taxa have endemism categories assigned on the master list, and (2) "Photographicrecord" should be two words: "Photographic record"

Once these are fixed, then I am happy for this to be deployed to the production site.

[helendallas]

@jeremyshelton @dimasciput I need to have FrogMap data on live site for the JRS final report for Odonata MAp.

Please can we aim to bring in the FrogMAP data to the production site as soon as Ryan has provided the BDI Links in our master list. I am hoping to get these by 22nd March.

@jeremyshelton Will you be able to upload the master list on the same day, or I can.

@dimasciput Can you find some time on the 24th to grab data.

@jeremyshelton I will let you know once I get updated master list from Ryan.

Thanks both!

[jeremyshelton]

@jeremyshelton @dimasciput I need to have FrogMap data on live site for the JRS final report for Odonata MAp.

@helendallas I have tested the FrogMap data on Testing Site. There are just two fixes needed for errors picked up on the csv: (1) Endemism column is blank, and (2) "Photographicrecord" should be two words: "Photographic record". I am waiting to hear back from @dimasciput about these fixes & then the FrogMAP data can go onto the Production Site.

Please can we aim to bring in the FrogMAP data to the production site as soon as Ryan has provided the BDI Links in our master list. I am hoping to get these by 22nd March.

@helendallas as per above. This should be a quick fix for @dimasciput

@jeremyshelton Will you be able to upload the master list on the same day, or I can.

@helendallas Im not sure I follow you here: the SA frog master list is already up on the Live Site? This is a list that we have worked closely with EWT and SANBI on.

@dimasciput Can you find some time on the 24th to grab data.

@jeremyshelton I will let you know once I get updated master list from Ryan.

@helendallas again, I am not quite sure what you mean here - I think Ryan needs to synchronise his pages with our SA species list? The FrogMAP list extends beyond SA and may have some species name inconsistencies with our master.

Thanks both!

[helendallas]

Sorry @jeremyshelton I forgot the anuran master list is already on live.

@dimasciput Do we need to upload the master list again to get the BDI links included?

I haven't yet received the list from Ryan, so I think once the two issue highlighted by Jeremy above, please go ahead and harvest from VM

[jeremyshelton]

@dimasciput is the VM data harvest now complete? Can I start testing this on the production site?

[jeremyshelton]

@dimasciput the VM frog data on production site are looking good now.

• Endemism is now fixed. Thanks.

• However, Photographicrecord still needs to be split into two words.

• If possible, could you also adjust the Source and Title columns to read: FrogMap Virtual Museum, FitzPatrick Institute of African Ornithology, University of Cape Town

[dimasciput]

However, Photographicrecord still needs to be split into two words.

This string is coming from the VM, I can fix the existing data but not for future data.

[helendallas]

Thanks @dimasciput I edited the Source Reference, easy for us to do @jeremyshelton . Closing ticket now as all looks ok.

[jeremyshelton]

Great. Thanks @helendallas