EarthScope / portable-fdsnws-dataselect

A portable fdsnws-dataselect server implementation
https://earthscope.github.io/portable-fdsnws-dataselect/
Apache License 2.0
7 stars 7 forks source link

Shipment logging patterns #1

Closed chad-earthscope closed 7 years ago

chad-earthscope commented 7 years ago

Shipment logging should be written to day-files in a user-specified directory. The shipment files should be named using the following pattern:

shipment-<UTCDate>

e.g. shipment-2017-04-25Z

Each connection should write a block in the following pattern:

START CLIENT <ClientHostName> [<ClientIP>] @ <UTCRequestTime> [<HTTP UserAgent>]
NET_STA_LOC_CHAN <Bytes>
NET_STA_LOC_CHAN <Bytes>
NET_STA_LOC_CHAN <Bytes>
...
END CLIENT <ClientHostName> [<ClientIP>] total bytes: <TotalBytes>

If the ClientHostName (a reverse lookup of the connected client address) is not available the ClientIP should be used. Obviously, these blocks should be written "atomically", i.e. not multiplexed with other blocks.

An example log snippet with two entries:

START CLIENT fw1.iris.washington.edu [128.95.166.29] @ 2017-04-25T17:04:38Z [Wget/1.12 (linux-gnu)]
IU_ANMO_00_LHZ 512
UW_COPS__SHZ 4096
TA_KMSC_EP_LDO 22528
END CLIENT fw1.iris.washington.edu [128.95.166.29] total bytes: 27136
START CLIENT 216.58.216.142 [216.58.216.142] @ 2017-04-25T17:04:38Z [IRISSeismic/1.4.5 RCurl/1.95-4.7 R/3.2.2 x86_64-pc-linux-gnu (MUSTANG)]
NL_HGN__LHZ 22528
IU_COLA_10_BHZ 5353472
UW_RAIN__SHZ 4096
END CLIENT 216.58.216.142 [216.58.216.142] total bytes: 5380096
chad-earthscope commented 7 years ago

Clarification: there should not be duplicate NET_STA_LOC_CHAN entries. The byte count for each NET_STA_LOC_CHAN should be a total for each NET_STA_LOC_CHAN.

chad-earthscope commented 7 years ago

Current formatting is wrong in the following ways:

  1. Header lines have malformed host name and request time:

    START CLIENT ('localhost', ['1.0.0.127.in-addr.arpa'], ['127.0.0.1']) [127.0.0.1] @ 06/May/2017 07:55:53 [curl/7.53.1]
  2. Footer lines have similarly malformed host name:

    END CLIENT ('localhost', ['1.0.0.127.in-addr.arpa'], ['127.0.0.1']) [127.0.0.1] total_bytes: 1077760
  3. Shipment information for the same NET_STA_LOC_CHAN is not combined, apparently being output per-section as identified by the index.