Making sure that configuration files are provided to the user for any marine region

[agruss2]

@QQ-Sortiz Today (June, 16th 2018), I queried OSMOSE parameters for the Gulf of Mexico to the web service. I did not modify any of the functional groups defined by the web service, did not change the order of the focal functional groups and biotic resources, and set the number of time steps per year to 12. I then reached the page "Download configuration files for your OSMOSE model" and pressed "Submit. Unfortunately, here is the message I got:

Generating configuration is not successful.
Click here to download the configuration JSON

Therefore, here is what I would like you to do, please: Could you please test the web service for all the marine ecosystems (i.e., Academician Berg, Adriatic Sea, etc.) and all the combinations of country/FAO area/state or province (i.e., United States (contiguous states)/Florida/Atlantic, Western Central) and make sure that, for all of the marine ecosystems and all the combinations of country/FAO area/state or province, the web service ultimately delivers a zip file, and not a JSON file? I suspect that this is solely a UI issue. Please let me know if this is not correct. Anyway, please let me know if what I reported here is clear enough and if you could address this important issue as soon as possible. Many thanks!

[agruss2]

@QQ-Sortiz @FIN-JBarile @Dengaloo I made some further tests this evening, and the web service delivered a JSON file rather than a zip file for ecosystems such as the Gulf of Mexico, the Adriatic Sea, or the Great Barrier Reef. I suspect that marine ecosystems characterized by a very high biodiversity entail a lot of computations, which the web service currently is not able to handle. Is there anything you think we could do to solve this issue? Many thanks!

[agruss2]

@QQ-Sortiz @FIN-JBarile @Dengaloo I made some new tests today, with the hope that this GitHub issue would have been resolved. However, there is a new issue now: whenever I request OSMOSE parameters for an ecosystem characterized by a large biodiversity such as the Gulf of Mexico, the Adriatic Sea, or the Great Barrier Reef, the following message appears when I press "Proceed" in Step 2:

Please select at least one background functional group.

Do you know what is causing that? Could all the issues raised in this comment and above be solved very soon, so that we can finalize the paper on the web service at long last? Many thanks!

[FIN-casey]

Fixed this problem. Tested with Academician Berg and Gulf of Mexico. Kindly check. Thanks!

[agruss2]

@QQ-Sortiz (CC: @FIN-JBarile @Dengaloo) I just ran a test where I queried parameters for the Gulf of Mexico. I deselected all the functional groups defined in the table of Step 2, except four of them and the "phytoplankton" and "zooplankton" groups. I was then able to obtain a zip file. Thus, here is the problem that we are facing: when the user queries OSMOSE parameter estimates for a marine region characterized by a high biodiversity, the table provided in Table 2 defines an extremely large number of functional groups, each potentially comprising many species. Then, the web service has to deal with an extremely large amount of information, this is cumbersome, and the web service therefore fails to provide OSMOSE configuration files within a zip file and provides a JSON file instead. These observations are encouraging, because they entail that there will be a way to deal with the present issue. However, what we would really want is to have a web service capable of handling an extremely large amount of information and of doing tons of computations in the background if need be. --> Thus, my question is: Is there any way that the web service could be enhanced to have the capacity to handle an extremely large amount of information and to do tons of computations in the background if need be, so as to ultimately be able to deliver a zip file for any marine region? We would want this, rather than putting additional constraints leading to the definition of a smaller number of functional groups comprising less species in Step 2. Please let me know what you think. Many thanks!

[FIN-JBarile]

@agruss2 cc: @Dengaloo

Apparently, it is not the large number of function groups/species that causes the issue. Instead, for the case of high biodiversity area where several functional groups are presented, there exist duplicates among functional groups. For example in the Gulf of Mexico, we have duplicates for fishbathydemersalmd, fishreefassociatedsm, invertreefassociatedsm, fishreefassociatedmd. This issue is addressed in the organization of the JSON data sources files of the UI.

@QQ-Sortiz will advise when correction is applied in the UI.

[QQ-Sortiz]

Hello @agruss2 cc: @Dengaloo @FIN-JBarile

Uploaded the new json files and was able to create a configuration file for Gulf of Mexico with all the functional groups checked. Kindly check, Thanks!

[agruss2]

@QQ-Sortiz @FIN-JBarile @Dengaloo Sorry for responding only know, but I needed to run a second series of tests on a second laptop at home, hence my late reply. This morning, I ran a first series of tests on my laptop at work, where I queried OSMOSE configuration files for the Gulf of Mexico to the web service, using two different web browsers (Google Chrome and Mozilla Firefox). Unfortunately, when using Google Chrome, when I pressed “Submit” in last step, instead of obtaining a zip file, the webpage “collapsed"; and, when using Mozilla Firefox, when I pressed “Submit” in last step, instead of obtaining a zip file, my laptop froze so that I had to "reset" it (i.e., restart it in an unsafe manner). When running the second series of tests on a second laptop at home, I obtained the same results when using Google Chrome and Mozilla Firefox. I really do not know what is going on. Especially because you indicated above that you managed to obtain a zip file for the Gulf of Mexico. This was without touching anything in Step 2, correct? Do you have any ideas why what I am describing above is happening to me? How could we please solve this issue to ensure that any user querying OSMOSE configuration files to the web service for any marine region from any laptop using any web browser always obtain a zip file when pressing “Submit” in last step? Many thanks!

[QQ-Sortiz]

Hello @agruss2 cc: @Dengaloo @FIN-JBarile

Running the wizard takes up the CPU process and memory allocation depending on the computer you are using. For example, we were able to generate a zip file for Gulf of Mexico on Firefox using my computer with a 4 core processor and 16gb ram and the running the application alone uses almost 8gb of memory. We also tried running it on another computer with 4 core processor and 8gb ram and was able to generate the zip file but the computer became sluggish because it uses almost 100% of the memory allocation. Tried to the third computer with 2 core processor and 4gb ram and the website crashes because it can't handle too much data processing of the application.

Tried also running it in Google Chrome on my computer and the website crashes because Chrome does not allow more than 4gb per tab of memory allocation. It was a security measure set by Google Chrome and the application uses approximately 8gb of memory when running the application for Gulf of Mexico that is why it crashes. You can check the memory allocation that it uses in Task Manager.

Kindly note that this only happen for regions with many functional groups and species available. The API is optimized but because of large amount of data being processed, it really takes a lot of memory.

One option is to create a local version of the wizard and API and run the application locally but it will still need the same memory allocation used. Another option is to define a limitation on a page that to be able to run a region with large amount of data like Gulf of Mexico, the user should have at least 8gb ram. (What is the likelihood of a user selecting all the functional groups and species for a very high biodiversity region and creating a configuration file?) Another one is to limit the number of functional groups selected if the region has a large amount of data. Lastly is a total overhaul of the application.

[jhpoelen]

@QQ-Sortiz from your description, it was not clear what the root cause of the crash is when generating a default osmose configuration after selecting the Gulf of Mexico. Also, why do you think that a "total overhaul of the application is needed"? And I was wondering what you mean by "the API is optimized". Do you mean the javascript code that runs in the browser? Or do you mean the web service that generates the osmose configuration zip file after receiving input from the wizard?

I'd be curious to hear @FIN-casey analysis/opinion about this especially because, judging from the commit log, she has done most, if not all, of the functional development of the wizard.

[jhpoelen]

@agruss2 @FIN-JBarile @Dengaloo - I just spent 2 hours on analyzing and preparing a fix for this issue. As I do not have write permissions to https://github.com/FIN-casey/FIN-casey.github.io/ , I prepared a pull request https://github.com/FIN-casey/FIN-casey.github.io/pull/2 with the fix in it. After careful analysis, it turned out that a relatively simple change avoided downloading and loading into memory >1GB worth of repeated data. After @FIN-casey merge the pull request, you should be able to confirm that this issue has now been resolved.

[agruss2]

This is wonderful ... I just made a series of tests where I queried OSMOSE configuration files for ecosystems characterized by a high biodiversity (the Gulf of Mexico, the Adriatic Sea, the Indian Ocean, and the Great Barrier Reef) and I was able to obtain a zip file within a reasonable time. This is making my week, so thank you very very much @jhpoelen ;) Oh ... and I can close the present issue now, then !