.from_string messes up column entries of wfdisc table

Gilaverbuch commented 8 months ago

Hi,

I am trying to read the wfdisc table entries that I generated in the CTBTO vDEC environment, but the .from_string function seems to mess up the columns entries. Via SQL, I generated a table in the required format:

FORMAT STRING: {0:6.6s} {1:8.8s} {2:17.5f} {3:8d} {4:8d} {5:8d} {6:17.5f} {7:8d} {8:11.7f} {9:16.6f} {10:16.6f} {11:6.6s} {12:1.1s} {13:2.2s} {14:1.1s} {15:64.64s} {16:32.32s} {17:10d} {18:8d} {19:%y-%m-%d %H:%M:%S}

CREATE TABLE wfdisc ( sta VARCHAR(6), chan VARCHAR(8), time FLOAT, wfid INTEGER NOT NULL, chanid INTEGER, jdate INTEGER, endtime FLOAT, nsamp INTEGER, samprate FLOAT, calib FLOAT, calper FLOAT, instype VARCHAR(6), segtype VARCHAR(1), datatype VARCHAR(2), clip VARCHAR(1), dir VARCHAR(64), dfile VARCHAR(32), foff INTEGER, commid INTEGER, lddate DATETIME NOT NULL, PRIMARY KEY (wfid), UNIQUE (wfid, dir, dfile) )

But when I parse the table to .from string it fails. to read it. I think that the problem is that the wfdisc data does not follow the expected input. Here is an example for a line: H01W1 EDH 1672545600 148771709 1117863 2023001 1672560000 3600000 250 543.72 .1 HITECH - s4 - /archive/ops/hydro/2023/001/ ArchH01W.1672531200.w 115200 -1 2023-01-05 01:03:02.000000000

For example, wfid expects 8 numbers, but it has 9.

Do you have any suggestions?

Thanks a lot, Gil

jkmacc-LANL commented 6 months ago

Hi @Gilaverbuch . I'm sorry for the delay! Thanks for the detailed Issue. There is a difference between CSS3.0 and KB CORE id lengths. Is is possible that you're using CSS3.0, but Pisces is expecting KB Core? Can you show how you created the Wfdisc class that's using .from_string?

Gilaverbuch commented 6 months ago

Hi @jkmacc-LANL. Thanks for your reply. I'm out of the office this week, so it may take me another week or so until I'll be able to check what you asked for. I'll get back to you as soon as I'm back. Thanks!

Gilaverbuch commented 6 months ago

Hi @jkmacc-LANL, Below is the code I am trying to use, and attached is the file:

class Wfdisc(css.Wfdisc):
    __tablename__ = 'Wfdisc'

wfdisc_file = 'H01W1.wfdisc'
wfdict = []
with open(wfdisc_file,'r') as ffwfdisc:
    for line in ffwfdisc:
        print(line)
        iwf = Wfdisc.from_string(
            line, default_on_error=['lddate']
            )
        print(iwf)

H01W1.csv

Thanks for your help

NINJA EDIT: code formatting

LANL-Seismoacoustics / pisces

.from_string messes up column entries of wfdisc table #77