search

ERDDAP / erddap

ERDDAP is a scientific data server that gives users a simple, consistent way to download subsets of gridded and tabular scientific datasets in common file formats and make graphs and maps. ERDDAP is a Free and Open Source (Apache and Apache-like) Java Servlet from NOAA NMFS SWFSC Environmental Research Division (ERD).
Creative Commons Zero v1.0 Universal
84 stars 58 forks source link

Change to using netcdf modules instead of netcdf-all #144

Closed ChrisJohnNOAA closed 5 months ago

ChrisJohnNOAA commented 5 months ago

Description

This loads netcdf dependencies as individual modules instead of the old way that needed to download netcdf-all outside of the normal dependency management.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

Checklist before requesting a review

srstsavage commented 5 months ago

Very happy to see this as the current wget approach in the pom wasn't working for me and I was just starting to put together some other possible approaches/band-aids. Obviously moving to the official set of dependency offerings from Unidata is the best path forward.

Tests pass for me locally (Linux, Maven 3.6.3, Java 17.0.3), with the exception of these which have already been identified in #140.

[ERROR] Errors:                                                                                          
[ERROR]   HashDigestTests.basicTest:21 » Runtime 
ERROR in Test.ensureEqual(Strings) line #1, col #1 '3[end]'!='9[end]':
Specifically, at line #1, col #1:
s1: 327fbb2aa6c6297d4fdd5fdf4b14e289[end]
s2: 96e9b4d974b734874a66f433b276a41a[end]
    ^

[ERROR]   NcHelperTests.testUnlimited:578 » FileNotFound /temp/unlimited.nc (No such file or directory)
[ERROR]   TableTests.testReadASCIISpeed:2558 Simple readASCII took too long (time=135ms > 130ms) (but often does when computer is busy).
[INFO] 
[ERROR] Tests run: 57, Failures: 0, Errors: 3, Skipped: 0

That's good enough for a :+1: from me!

BobSimons commented 5 months ago

I tried to do this a few times and failed each time. See the commented-out section of pom.xml. Perhaps you succeeded where I failed, but... I worked with the netcdf-java developers but they couldn't give me a list of modules to mimic netcdf-all and couldn't resolve the other problems. Getting it to compile is no problem. Getting it to pass all of the tests is the problem. (Remember that netcdf-java is responsible for reading many file types beyond .nc and .hdf.) I don't think anyone has all of the tests except Chris, and many of the tests are not working now because of the RAID failure last spring making several datasets permanently unavailable. I recommended that fixing those tests (by using new datasets) was a very high priority but was ignored.

On Thu, Apr 11, 2024 at 1:56 PM Chris John @.***> wrote:

Description

This loads netcdf dependencies as individual modules instead of the old way that needed to download netcdf-all outside of the normal dependency management.

Fixes # (issue) Type of change

Please delete options that are not relevant.

  • [ X] Bug fix (non-breaking change which fixes an issue)

Checklist before requesting a review

  • [ X] I have performed a self-review of my code
  • [ X] My code follows the style guidelines of this project
  • [X ] I have commented my code, particularly in hard-to-understand areas
  • [ X] I have made corresponding changes to the documentation
  • [ X] My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • [X ] New and existing unit tests pass locally with my changes

You can view, comment on, or merge this pull request online at:

https://github.com/ERDDAP/erddap/pull/144 Commit Summary

File Changes

(1 file https://github.com/ERDDAP/erddap/pull/144/files)

Patch Links:

— Reply to this email directly, view it on GitHub https://github.com/ERDDAP/erddap/pull/144, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALKWOCUDMVZGGTCR535SHLY43FFNAVCNFSM6AAAAABGCX76KWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIZTQMRVGQ2DKMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

mwengren commented 5 months ago

@ChrisJohnNOAA One dependency that may be missing is an SLF4J implmentation.

When I run ERDDAP locally, I'm getting the following SLF4J warning messages now:

////**** EDStatic Low Level Startup
localTime=2024-04-12T15:35:41-04:00
erddapVersion=2.23
Java 18.0.2.1 (64 bit, Oracle Corporation) on Linux (4.15.0-221-generic).
MemoryInUse=   133 MB (highWaterMark=   133 MB) (Xmx ~= 2048 MB)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
logLevel=info: verbose=true reallyVerbose=false
%%% TouchThread: new touchThread started at 2024-04-12T15:35:41-04:00
ERROR in File2.deleteIfOld: dir=data/dataset/_FileVisitor/ isn't a directory.
bigParentDirectory=data/
...

Not sure if this is a real issue or not (ERDDAP is running for me), but SLF4J is mentioned as a separate dependency that should be included in the Unidata docs for netcdf-Java.

I don't see an SLF4j dependency included anywhere in the current pom.xml.

ChrisJohnNOAA commented 5 months ago

@mwengren I have a fix that here: https://github.com/ERDDAP/erddap/pull/146