search

osmose-model / osmose-web-api

Web service that generates Osmose configuration files from data sources like Fishbase and SeaLifeBase. Used by https://www.config.osmose-model.org .
MIT License
2 stars 2 forks source link

missing fishbase_archiver v0.2.1-patch4/table_names.tsv #176

Closed jhpoelen closed 5 years ago

jhpoelen commented 5 years ago

When checking the server logs to understand @agruss2 comment (https://github.com/jhpoelen/fb-osmose-bridge/issues/175#issuecomment-417822673), I found that a recent patch release of the fishbase/sealifebase archives was missing the file "table_names.tsv". This file indicates which tables are included in the release and is expected to be present for each release. It turned out that a compressed version table_names.tsv.gz was included. I have corrected this (minor) issue by replacing the table_names.tsv.gz with its uncompressed version table_names.tsv.

@FIN-JBarile @QQ-Sortiz : thanks for using the patch releases on https://github.com/jhpoelen/fishbase_archive to provide fast access to up-to-date fishbase/sealifebase data. For future (patch) releases, please use a file with name table_names.tsv to document the patched/released tables.

Associated server logs:

2018-09-01T01:09:57.218272+00:00 app[web.1]: Sep 01, 2018 1:09:57 AM com.github.jhpoelen.fbob.ValueFactoryCache availableTables
2018-09-01T01:09:57.218285+00:00 app[web.1]: SEVERE: failed to retieve tables
2018-09-01T01:09:57.218288+00:00 app[web.1]: java.io.FileNotFoundException: https://github.com/jhpoelen/fishbase_archiver/releases/download/v0.2.1-patch4/table_names.tsv
2018-09-01T01:09:57.218289+00:00 app[web.1]:    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1890)
2018-09-01T01:09:57.218291+00:00 app[web.1]:    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
2018-09-01T01:09:57.218293+00:00 app[web.1]:    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
2018-09-01T01:09:57.218295+00:00 app[web.1]:    at java.net.URL.openStream(URL.java:1045)
2018-09-01T01:09:57.218297+00:00 app[web.1]:    at org.apache.commons.io.IOUtils.toString(IOUtils.java:894)
2018-09-01T01:09:57.218299+00:00 app[web.1]:    at org.apache.commons.io.IOUtils.toString(IOUtils.java:848)
2018-09-01T01:09:57.218301+00:00 app[web.1]:    at org.apache.commons.io.IOUtils.toString(IOUtils.java:833)
2018-09-01T01:09:57.218302+00:00 app[web.1]:    at com.github.jhpoelen.fbob.ValueFactoryCache.availableTables(ValueFactoryCache.java:203)
jhpoelen commented 5 years ago

A separate, yet related issue I found is related to the encoding and newline of the table_names.tsv. Please use the unix-style newline instead of the DOS-style newline in the future to be consistent with existing data releases. See https://en.wikipedia.org/wiki/Newline#Issues_with_different_newline_formats .

jhpoelen commented 5 years ago

After applying described fixes, I found that the osmose web api is now using the patch4 for both fishbase and sealifebase tables:

2018-09-01T17:32:07.539565+00:00 app[web.1]: [https://github.com/jhpoelen/fishbase_archiver/releases/download/v0.2.1-patch4/poplw_fishbase.tsv.gz] downloading...
2018-09-01T17:32:07.897321+00:00 app[web.1]: [https://github.com/jhpoelen/fishbase_archiver/releases/download/v0.2.1-patch4/poplw_fishbase.tsv.gz] downloaded in [357] ms.
2018-09-01T17:32:10.545139+00:00 app[web.1]: [https://github.com/jhpoelen/fishbase_archiver/releases/download/v0.2.1/poplw_fishbase.tsv.gz] downloading...
2018-09-01T17:32:11.152007+00:00 app[web.1]: [https://github.com/jhpoelen/fishbase_archiver/releases/download/v0.2.1/poplw_fishbase.tsv.gz] downloaded in [607] ms.
2018-09-01T17:32:14.868342+00:00 app[web.1]: [https://github.com/jhpoelen/fishbase_archiver/releases/download/v0.2.1-patch4/poplw_sealifebase.tsv.gz] downloading...
2018-09-01T17:32:15.064840+00:00 app[web.1]: [https://github.com/jhpoelen/fishbase_archiver/releases/download/v0.2.1-patch4/poplw_sealifebase.tsv.gz] downloaded in [196] ms.
2018-09-01T17:32:15.158145+00:00 app[web.1]: [https://github.com/jhpoelen/fishbase_archiver/releases/download/v0.2.1/poplw_sealifebase.tsv.gz] downloading...
2018-09-01T17:32:15.394192+00:00 app[web.1]: [https://github.com/jhpoelen/fishbase_archiver/releases/download/v0.2.1/poplw_sealifebase.tsv.gz] downloaded in [243] ms.