search

globalbioticinteractions / nomer

maps identifiers and names to other identifiers and names
GNU General Public License v3.0
19 stars 3 forks source link

warnings generated when matching against discoverlife-taxon #44

Closed jhpoelen closed 2 years ago

jhpoelen commented 3 years ago

$ java -version java version "13" 2019-09-17 Java(TM) SE Runtime Environment (build 13+33) Java HotSpot(TM) 64-Bit Server VM (build 13+33, mixed mode, sharing)

$ echo -e "\tDonald duck" | nomer append discoverlife-taxon using matcher [discoverlife-taxon] DiscoverLife name indexing started... WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.mapdb.SerializerPojo$FieldInfo (file:/usr/local/bin/nomer) to field java.util.Collections$UnmodifiableMap.m WARNING: Please consider reporting this to the maintainers of org.mapdb.SerializerPojo$FieldInfo WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [50590] DiscoverLife names were indexed in 39s (@ 1297 names/s) Donald duck NONE Donald duck

on

Darwin XXXXX.local 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:31 PDT 2021; root:xnu-7195.141.2~5/RELEASE_X86_64 x86_64

jhpoelen commented 3 years ago

Root cause for the warning lies in improper use of access modifiers in mapdb.

Warning can be silenced, but best is to get another version of mapdb or replacement library.

jhpoelen commented 3 years ago

Note that these warning are specific to Java v13.

jhpoelen commented 3 years ago

I've added the following text to the to the prerequisites section of the Nomer readme:

Nomer needs Java 8+, and is developed and tested on OpenJDK 8. Why OpenJDK 8?. Because OpenJDK 8 is Long Term Supported (LTS), and (at time of writing Nov 2021) supported until at least May 2026, longer than any of the newer OpenJDK versions.

jhpoelen commented 2 years ago

see also #103

jhpoelen commented 2 years ago

Currently, the globi libs generate 18 related warnings in the continuous integration build. see attached screenshot.

Screenshot from 2022-09-13 15-01-38

jhpoelen commented 2 years ago

after upgrade to mockito 3.12.4, the number of warnings reduced to 9.

Screenshot from 2022-09-13 15-07-08

jhpoelen commented 2 years ago

I was able to reproduce the warning on openjdk11

$ java -version
openjdk version "11.0.16" 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu122.04, mixed mode, sharing)
$ echo -e "\tQuercus" | nomer append wfo
[main] INFO org.globalbioticinteractions.nomer.match.TermMatcherRegistry - using matcher [wfo]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.mapdb.SerializerPojo$FieldInfo WARNING: Please consider reporting this to the maintainers of org.mapdb.SerializerPojo$FieldInfo
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[main] INFO org.globalbioticinteractions.nomer.match.WorldOfFloraOnlineTaxonService - 
    Quercus HAS_ACCEPTED_NAME   WFO:4000032377  Quercus genus       Angiosperms | Fagales | Fagaceae | Quercus  WFO:9949999999 | WFO:9000000208 | WFO:7000000231 | WFO:4000032377   phylum | order | family | genus http://www.worldfloraonline.org/taxon/wfo-4000032377
jhpoelen commented 2 years ago

after reading @jankotek 's comment in - https://github.com/jankotek/mapdb/issues/203#issuecomment-40769515

I attempted to explicitly set key/value serializer to avoid the default serializer (i.e., SerializerPojo) to be used.

Also, I disable application of some mmap "hack" to workaround https://github.com/jankotek/mapdb/issues/442 and http://bugs.java.com/view_bug.do?bug_id=4724038

also see https://github.com/jankotek/mapdb/blob/5ef38a982a04fbd8e5e4714d5474b698921cbb74/src/main/java/org/mapdb/DBMaker.java#L391

jhpoelen commented 2 years ago

@jtmiller28 After applying changes introduced via https://github.com/globalbioticinteractions/nomer/commit/6c9ec8d5ebe0ec334baedaec20109820aadb781b , I was unable to reproduce the warnings using java openjdk11 . This suggests that the issue was been resolved.

Thanks for reminding me about this funky issue, hoping to release a maintenance version of Nomer to help reduce the scary looking output on terminal.

jtmiller28 commented 2 years ago

Hey Jorrit, thanks for looking into this! The fix isn't live in Nomer and projected for the next update right? I updated my Nomer and did a test, and still received the same warnings generated by Java.

jhpoelen commented 2 years ago

https://github.com/globalbioticinteractions/nomer/releases/tag/0.3.1 is now available with fix for this issue. @jtmiller28 please review.

jtmiller28 commented 2 years ago

I think I am still running into the same issue: nomer version 0.3.1 echo -e "\tEchinocereus engelmannii subsp. engelmannii" | nomer append wfo [main] INFO org.globalbioticinteractions.nomer.match.TermMatcherRegistry - using matcher [wfo] WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.mapdb.SerializerPojo$FieldInfo (file:/usr/local/bin/nomer) to field java.util.Collections$SingletonList.element WARNING: Please consider reporting this to the maintainers of org.mapdb.SerializerPojo$FieldInfo WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [main] INFO org.globalbioticinteractions.nomer.match.WorldOfFloraOnlineTaxonService - [WORLD_OF_FLORA_ONLINE] taxonomy already indexed at [/home/jt-miller/.cache/nomer/world_of_flora_online/world_of_flora_online], no need to import. Echinocereus engelmannii subsp. engelmannii HAS_ACCEPTED_NAME WFO:0001430711 Echinocereus engelmannii subsp. engelmannii subspecies Angiosperms | Caryophyllales | Cactaceae | Echinocereus | Echinocereus engelmannii | Echinocereus engelmannii subsp. engelmannii WFO:9949999999 | WFO:9000000088 | WFO:7000000098 | WFO:4000012914 | WFO:0000661245 | WFO:0001430711 phylum | order | family | genus | species | subspecies http://www.worldfloraonline.org/taxon/wfo-0001430711
java --version openjdk 11.0.16 2022-07-19 OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu122.04) OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu122.04, mixed mode, sharing)

jtmiller28 commented 2 years ago

Though also note that it doesnt generate for these warnings for your example: echo -e "\tDonald duck" | nomer append discoverlife-taxon [main] INFO org.globalbioticinteractions.nomer.match.TermMatcherRegistry - using matcher [discoverlife-taxon] Donald duck NONE Donald duck

Looking into to it, could it be catalogue dependent? echo -e "\tDonald duck" | nomer append itis [main] INFO org.globalbioticinteractions.nomer.match.ITISTaxonService - [ITIS] taxonomy imported. Donald duck NONE Donald duck

WFO is the only catalogue of the ones I've tested to trip the warning messages: echo -e "\tDonald duck" | nomer append wfo [main] INFO org.globalbioticinteractions.nomer.match.TermMatcherRegistry - using matcher [wfo] WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.mapdb.SerializerPojo$FieldInfo (file:/usr/local/bin/nomer) to field java.util.Collections$SingletonList.element WARNING: Please consider reporting this to the maintainers of org.mapdb.SerializerPojo$FieldInfo WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [main] INFO org.globalbioticinteractions.nomer.match.WorldOfFloraOnlineTaxonService - [WORLD_OF_FLORA_ONLINE] taxonomy already indexed at [/home/jt-miller/.cache/nomer/world_of_flora_online/world_of_flora_online], no need to import. Donald duck NONE Donald duck

jhpoelen commented 2 years ago

@jtmiller28 thanks for the detailed messages.

A cache built with a prior version (< v0.3.1) is expected to generate the same warning messages that we are trying to get rid of.

Can you please run:

nomer clean

to flush the cache, then do

echo -e "\tQuercus" | nomer append wfo

The reason for this is that the source of the warning was the kind of serializer used to save data to disk. I switched the serializer to a friendlier one, but an old cache would still generate the message, because the cache uses information encoded in the data to use the same serializer / deserializer used to encoded the cache.

Apologies for all this, , , caching has great benefits and also introduces some side effects.

Curious to see whether the nomer clean trick does the job for you.

jhpoelen commented 2 years ago

@jtmiller28 I just reproduced the warning by building the cache with v0.3.0 and using it with v0.3.1 -

build cache with v0.3.0

$ echo -e "\tQuercus" | java -jar nomer.jar append wfo
[main] INFO org.globalbioticinteractions.nomer.match.TermMatcherRegistry - using matcher [wfo]
[main] INFO org.globalbioticinteractions.nomer.match.WorldOfFloraOnlineTaxonService - [WORLD_OF_FLORA_ONLINE] taxonomy importing...
[main] INFO org.globalbioticinteractions.nomer.match.ResourceServiceContentBased - using local Preston data dir: [/home/jorrit/.cache/nomer/data]
[main] INFO org.globalbioticinteractions.nomer.match.ResourceServiceContentBased - caching [zip:http://104.198.143.165/files/WFO_Backbone/_WFOCompleteBackbone/WFO_Backbone.zip!/classification.txt] at [/home/jorrit/.cache/nomer/tmp/nomer9105818633497411630.gz]...
[https://zenodo.org/recor...210ef59e06b640a3539cb5a] 100.0% of 78 bytes at 0.01 MB/s completed in < 1 minute
[https://zenodo.org/recor...eac1915f7c3be9748eda991] 100.0% of 22 kB at 1.28 MB/s completed in < 1 minute
[https://zenodo.org/recor...dd472e8ae9c3f66f4932c62] 100.0% of 78 bytes at 0.07 MB/s completed in < 1 minute
[https://zenodo.org/recor...4e871ad1ec0a8b283091e08] 100.0% of 26 kB at 3.74 MB/s completed in < 1 minute
[https://zenodo.org/recor...ac9a12e14bfa72ccb0a6828] 100.0% of 78 bytes at 0.04 MB/s completed in < 1 minute
[https://zenodo.org/recor...f20e2eedbb2fb585bdf0822] 100.0% of 32 kB at 15.91 MB/s completed in < 1 minute
[https://zenodo.org/recor...5bba06ad7b40eea7c8c9831] 100.0% of 78 bytes at 0.07 MB/s completed in < 1 minute
[https://zenodo.org/recor...ae02993ebf7a0fe30d87137] 100.0% of 2 kB at 1.32 MB/s completed in < 1 minute
[https://zenodo.org/recor...0166e3d10ee520b18336b09] 100.0% of 78 bytes at 0.07 MB/s completed in < 1 minute
[https://zenodo.org/recor...9ab1999577ca5d085b382a3] 100.0% of 3 kB at 1.94 MB/s completed in < 1 minute
[https://zenodo.org/recor...9cc9e855907e516dd7de8a7] 100.0% of 78 bytes at 0.07 MB/s completed in < 1 minute
[https://zenodo.org/recor...000c74e4e8fdeb937e29b1d] 100.0% of 34 kB at 11.08 MB/s completed in < 1 minute
[https://zenodo.org/recor...3c6dadeb82d44a60c7552fb] 100.0% of 89 MB at 5.14 MB/s completed in < 1 minute
[main] INFO org.globalbioticinteractions.nomer.match.ResourceServiceContentBased - caching [zip:http://104.198.143.165/files/WFO_Backbone/_WFOCompleteBackbone/WFO_Backbone.zip!/classification.txt] at [/home/jorrit/.cache/nomer/tmp/nomer9105818633497411630.gz] done.
[main] INFO org.globalbioticinteractions.nomer.match.ResourceServiceReadOnly - using cached [zip:http://104.198.143.165/files/WFO_Backbone/_WFOCompleteBackbone/WFO_Backbone.zip!/classification.txt] at [/home/jorrit/.cache/nomer/hash/sha256/f4e2b9806440d0605f60b81feb9782655291aac2d000c74e4e8fdeb937e29b1d/688b7ec7253fd205e2d3c5e83d05bcf889d6cc5cd1d3e7ec7841217df82f9956.gz]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.mapdb.SerializerPojo$FieldInfo (file:/home/jorrit/proj/globi/nomer/nomer.jar) to field java.util.Collections$SingletonList.element
WARNING: Please consider reporting this to the maintainers of org.mapdb.SerializerPojo$FieldInfo
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[main] INFO org.globalbioticinteractions.nomer.match.WorldOfFloraOnlineTaxonService - cache with [1425061] items built in [232.2] s or [6136.9] items/s.
[main] INFO org.globalbioticinteractions.nomer.match.WorldOfFloraOnlineTaxonService - [WORLD_OF_FLORA_ONLINE] taxonomy imported.
    Quercus HAS_ACCEPTED_NAME   WFO:4000032377  Quercus genus       Angiosperms | Fagales | Fagaceae | Quercus  WFO:9949999999 | WFO:9000000208 | WFO:7000000231 | WFO:4000032377   phylum | order | family | genus http://www.worldfloraonline.org/taxon/wfo-4000032377

now use it with v0.3.1 

$ echo -e "\tQuercus" | nomer append wfo
[main] INFO org.globalbioticinteractions.nomer.match.TermMatcherRegistry - using matcher [wfo]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.mapdb.SerializerPojo$FieldInfo (file:/home/jorrit/proj/globi/nomer/nomer/target/nomer-0.0.1-SNAPSHOT-jar-with-dependencies.jar) to field java.util.Collections$SingletonList.element
WARNING: Please consider reporting this to the maintainers of org.mapdb.SerializerPojo$FieldInfo
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[main] INFO org.globalbioticinteractions.nomer.match.WorldOfFloraOnlineTaxonService - [WORLD_OF_FLORA_ONLINE] taxonomy already indexed at [/home/jorrit/.cache/nomer/world_of_flora_online/world_of_flora_online], no need to import.
    Quercus HAS_ACCEPTED_NAME   WFO:4000032377  Quercus genus       Angiosperms | Fagales | Fagaceae | Quercus  WFO:9949999999 | WFO:9000000208 | WFO:7000000231 | WFO:4000032377   phylum | order | family | genus http://www.worldfloraonline.org/taxon/wfo-4000032377

on clearing the cache and using v0.3.1 to rebuild cache, the warning is no longer observed.

Please let me know if you see the same thing.

jtmiller28 commented 2 years ago

Clearing the cache indeed does the trick: echo -e "\tQuercus" | nomer append wfo [main] INFO org.globalbioticinteractions.nomer.match.TermMatcherRegistry - using matcher [wfo] [main] INFO org.globalbioticinteractions.nomer.match.WorldOfFloraOnlineTaxonService - [WORLD_OF_FLORA_ONLINE] taxonomy already indexed at [/home/jt-miller/.cache/nomer/world_of_flora_online/world_of_flora_online], no need to import. Quercus HAS_ACCEPTED_NAME WFO:4000032377 Quercus genus Angiosperms | Fagales | Fagaceae | Quercus WFO:9949999999 | WFO:9000000208 | WFO:7000000231 | WFO:4000032377 phylum | order | family | genus http://www.worldfloraonline.org/taxon/wfo-4000032377

Thanks for attention to these issues!

jhpoelen commented 2 years ago

@jtmiller28 great! Are you ok closing this issue?

jtmiller28 commented 2 years ago

Yep! I can’t close it since it’s not my open issue(?), but the update fulfills all of my concerns with that particular warning.

jhpoelen commented 2 years ago

Good to hear! Closing issue.

btw - if you'd like to get more permissions on the nomer repo, so you can manage issues etc, please let me know!