search

USF-IMARS / erddap-config

Content dir for docker-erddap incl setup.xml & dataset.xml
0 stars 1 forks source link

sat images silently failing #16

Closed 7yl4r closed 3 years ago

7yl4r commented 3 years ago

Images on the FGBNMS dash are out. When trying to use the griddap interface directly (eg), the image never loads.

Looking at status.html I see:

n Datasets Failed To Load (in the last major LoadDatasets) = 1
    chlor_a_l3_pass_db6d_37b4_723b, (end)
Response Failed    Time (since last major LoadDatasets) n =       26,  median ~=  6900000 ms
Response Failed    Time (since last Daily Report)       n =       26,  median ~=  6900000 ms
Response Failed    Time (since startup)                 n =       75,  median ~=        4 ms
Response Succeeded Time (since last major LoadDatasets) n =      231,  median ~=        2 ms
Response Succeeded Time (since last Daily Report)       n =      236,  median ~=        2 ms
Response Succeeded Time (since startup)                 n =     7375,  median ~=      105 ms
TaskThread has finished 0 out of 0 tasks.  Currently, no task is running.
TaskThread Failed    Time (since last Daily Report)     n =        0
TaskThread Failed    Time (since startup)               n =        0
TaskThread Succeeded Time (since last Daily Report)     n =        0
TaskThread Succeeded Time (since startup)               n =        0
Number of threads: Tomcat-waiting=0, inotify=13, other=198
MemoryInUse=  1753 MB (highWaterMark=  1856 MB) (Xmx ~= 65303 MB)

We have some errors in docker logs erddap. Most recent one below:

[...]
25-Oct-2021 17:16:57.847 INFO [http-nio-8080-exec-940] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
 Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
        java.lang.IllegalArgumentException: Invalid character found in method name [0x160x030x010x010xfe0x010x000x010xfa0x030x030xbcV0x0d0xc50xc50x93,0x9b0xe170xab0x090x1e0xb9z0x13<3n60xf80xeaE0xb6M50xcd0xa10x010xff0xe3Y0x000x01<0xcc0x140xcc0x130xcc0x150xc000xc0,0xc0(0xc0$0xc00x140xc00x0a0x000xa50x000xa30x000xa10x000x9f0x00k0x00j0x00i0x00h0x0090x0080x0070x0060xc0w0xc0s0x000xc40x000xc30x000xc20x000xc10x000x880x000x870x000x860x000x850xc00x190x000xa70x00m0x00:0x000xc50x000x890xc020xc0.0xc0*0xc0&0xc00x0f0xc00x050xc0y0xc0u0x000x9d0x00=0x0050x000xc00x000x840x000x950xc0/0xc0+0xc0'0xc0#0xc00x130xc00x090x000xa40x000xa20x000xa00x000x9e0x00g0x00@0x00?0x00>0x0030x0020x0010x0000xc0v0xc0r0x000xbe0x000xbd0x000xbc0x000xbb0x000x9a0x000x990x000x980x000x970x00E0x00D0x00C0x00B0xc00x180x000xa60x00l0x0040x000xbf0x000x9b0x00F0xc010xc0-0xc0)0xc0%0xc00x0e0xc00x040xc0x0xc0t0x000x9c0x00<0x00/0x000xba0x000x960x00A0x000x070x000x940xc00x110xc00x070x00f0xc00x160x000x180xc00x0c0xc00x020x000x050x000x040x000x920xc00x120xc00x080x000x160x000x130x000x100x000x0d0xc00x170x000x1b0xc00x0d0xc00x030x000x0a0x000x930x00c0x000x150x000x120x000x0f0x000x0c0x000x1a0x00b0x000x090x00a0x00e0x00d0x00`0x000x140x000x110x000x0e0x000x0b0x000x190x000x080x000x060x000x170x000x030xc00x150xc00x100xc00x060xc00x0b0xc00x010x00;0x000x020x000x010x000xff0x020x010x000x000x940x000x000x000x140x000x120x000x000x0f131.247.136.2000x000x0b0x000x040x030x000x010x020x000x0a0x00:0x0080x000x0e0x000x0d0x000x190x000x1c0x000x0b0x000x0c0x000x1b0x000x180x000x090x000x0a0x000x1a0x000x160x000x170x000x080x000x060x000x070x000x140x000x150x000x040x000x050x000x120x000x130x000x010x000x020x000x030x000x0f0x000x100x000x110x00#0x000x000x000x0d0x00]. HTTP method names must be tokens
                at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:431)
                at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:503)
                at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
                at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1634)
                at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.lang.Thread.run(Thread.java:748)
7yl4r commented 3 years ago

Looking at the underlying data I found that ls on the following directories took a long time (~30s) to mount:

It looks like those are the only two from thing2 so there is some work to be done there. I don't know if that is the cause of this issue however.

7yl4r commented 3 years ago

The docker container shows as having been up for 3 months. We definitely haven't had this issue that long.

[root@dune ~]# docker container ls
CONTAINER ID   IMAGE                 COMMAND                  CREATED        STATUS        PORTS                                       NAMES
5bdcd91cfd9a   axiom/docker-erddap   "/entrypoint.sh cata…"   3 months ago   Up 3 months   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   erddap

Wait... didn't I already try resetting things? 3mo can't be right. Let's try doing some resets again.

[root@dune ~]# docker container ls
CONTAINER ID   IMAGE                 COMMAND                  CREATED        STATUS        PORTS                                       NAMES
5bdcd91cfd9a   axiom/docker-erddap   "/entrypoint.sh cata…"   3 months ago   Up 3 months   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   erddap
[root@dune ~]# docker restart erddap
erddap
[root@dune ~]# docker container ls
CONTAINER ID   IMAGE                 COMMAND                  CREATED        STATUS          PORTS                                       NAMES
5bdcd91cfd9a   axiom/docker-erddap   "/entrypoint.sh cata…"   3 months ago   Up 14 seconds   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   erddap

aaaaand now we wait for ERDDAP to do its thing for a few hours.

7yl4r commented 3 years ago

it working! yay for turning it off and on again!