search

kenfar / DataGristle

Tough and flexible tools for data analysis, transformation, validation and movement.
Other
136 stars 18 forks source link

Improve pipe handling #12

Open kenfar opened 12 years ago

kenfar commented 12 years ago

Received this error:

kenfar@glenda:~/projects/datagristle/scripts$ ./gristle_slicer.py ../../data_general/2012* -c 0:10 -r 0:50 | ./gristle_viewer.py -r 33 missing delimiter - will abort Usage: The gristle_viewer.py is used to view data one record at a time. gristle_viewer.py [file] -q -v -h --delimiter [value] --recnum [value]

gristle_viewer.py: error: Please provide delimiter when piping data into program via stdin or reading multiple input files close failed in file object destructor: Error in sys.excepthook:

Original exception was:

References to this error: http://stackoverflow.com/questions/180095/how-to-handle-a-broken-pipe-sigpipe-in-python http://stackoverflow.com/questions/6537178/python-how-to-catch-a-broken-pipe

This appears to be a problem with the second program closing while the first is still sending data. A try/except block in the first didn't catch the error. Not sure what to do here.

kenfar commented 2 years ago

Fixed this in gristle_slicer - by just looping through the input data but ignoring the recs once I have the final rec I need.

Need to apply this approach to more of the programs.