search

osmose-model / osmose-web-api

Web service that generates Osmose configuration files from data sources like Fishbase and SeaLifeBase. Used by https://www.config.osmose-model.org .
MIT License
2 stars 2 forks source link

UI Step 2 - Issue no. 12 #106

Closed agruss2 closed 7 years ago

agruss2 commented 7 years ago

@FIN-casey @FIN-JBarile Please make sure that invertebrate functional groups defined by SeaLifeBase are also provided in the table.

FIN-JBarile commented 7 years ago

@FIN-casey will apply soonest. She has necessary materials to proceed with this. Thanks.

FIN-casey commented 7 years ago

Hi I added the invertebrate functional groups defined by SeaLifeBase in the data file, but the data file for Country+FAO combination exceeds 100mb, which is the maximum file size for Git. The other two data files (Country+SubCountry+FAO and Ecosystem) are already commited. They are included in the table.

To add the large data file, we need to look for other storage area that the UI can link to. It can be a website. Can we use fishbase website for this?

jhpoelen commented 7 years ago

@FIN-casey one way I can think of to work around this limitation is to the big files into separate files in a folder (grouped by country code (C_Code). This might also speed up the retrieval, because you don't have to plow through a massive json dump.

Example: instead of single file countryFAO_funcgrp.json

introduce files: countryFAO_funcgrp/countryFAO_funcgrp_xyz.json countryFAO_funcgrp/countryFAO_funcgrp_abc.json countryFAO_funcgrp/countryFAO_funcgrp_def.json ...

where abc, def, xyz are country codes (C_Codes).

Other grouping can be used when suitable. Curious to hear how you are going to address this.

FIN-JBarile commented 7 years ago

@jhpoelen thanks for your suggestion.

Apologies for I missed to filter only for inverts.

@agruss2 Are the groups below only the ones of interest for this purpose? Please advise. Thanks.

TargetGrpNum TargetGrp Name Description 25 33554432 cephalopods cephalopods 26 67108864 shrimp shrimp 27 134217728 lobsters,crab lobsters,crab 28 268435456 jellyfish jellyfish 29 536870912 demersal mollusc demersal mollusc 30 1073741824 krill krill

agruss2 commented 7 years ago

@FIN-JBarile The list that you provided is too restrictive to my mind. I thought that SeaLifeBase had data/parameters for a much larger number of non-fish organisms. I was thinking that the bridge could also provide data for octopods, as well as for a diversity of low-trophic level marine organisms, such as echinoderms, Ascidiacea spp., Brachiopoda spp., Bryozoa spp., Crinoidea spp., Hydrozoa spp., Ophiuroidea spp., Polychaeta spp., Amphipoda spp., Isopoda spp., Mysidacea spp., Ostracoda spp., Turbellaria spp., Nematoda spp., zooplankton species, phytoplankton species, etc. OSMOSE users would need paramaters for the abovementioned low-trophic level marine organisms, so as to be able to define a reasonable number of background functional groups for their OSMOSE model.

--> Is it doable to obtain data/parameters from SeaLifeBase for low-trophic level marine organisms such as the ones I mentioned above? Many thanks. -----> @agruss2 Yes, it is possible. With your clarification, I just now have to remove the Chordata species. You will have to inspect data richness in terms of your definition in #59 though. Dataset is in NonFish_DataProfile.zip. You may also want to consider filtering for data richness. Thanks.

agruss2 commented 7 years ago

@FIN-JBarile Many thanks; I had a look at the zip file. I confirm that, for invertebrates, we should apply the exact same rules for data richness as described in #59. This is because, by default, the bridge between FishBase/SeaLifeBase and OSMOSE defines all functional groups as "focal functional groups"; then, the user can choose to redefine some of the functional groups as "background functional groups".

FIN-casey commented 7 years ago

The invertebrate functional groups are now included in the data set of the UI. The order is based on data richness. Please verify so we can close the issue. THanks!

FIN-casey commented 7 years ago

Hi @jhpoelen thanks for the suggestion, but if I separate the files by Country, there will arise a problem in searching for class or species because the API searches on all the dataset. What I did is I split the CountryFAO file into 2 json files.

agruss2 commented 7 years ago

@FIN-casey @FIN-JBarile Many thanks for having implemented this! I am going to close the present issue, because what needed to be addressed here has been addressed. However, I am looking at the config files now and am noticing a number of issues with the invertebrates, even when those are defined as background functional groups (and, therefore, require less parameters); the issue that we have to address now is that the great majority of invertebrates are extremely data-poor. What I am going to do is that I am going to review the config files in more detail; then, later this week, I will create a new GitHub issue describing what I think we should do to address this problem of data scarcity with the great majority of the invertebrates, which result in too many parameters being set at unrealistic default values. Talk soon, and thanks a lot once again!