pavlis
commented
2 years ago I'm not sure if these problems are linked, but I have now had mongodb crash on me twice running a parallel version of the job that creates the very verbose messages that are the topic of this issues.
What I'm trying to do is post computed P wave travel times to a year of lower 48 broadband data from 2012. MsPASS team that is our larger test data set. I was working on a full up test of a workflow to process this full year of data on the old mac pro in my office. So, I was running mspass in the docker container. FYI, this is the python code that has crashed mongodb twice now:
from dask import bag
from dask.distributed import Client
import time
client = Client()
# could use a different earth model here but probably a small detail
model = TauPyModel(model="iasp91")
stimeutc=UTCDateTime('2012-01-01T00:00:00.0')
etimeutc=UTCDateTime('2013-01-01T00:00:00.0')
query={'starttime' : {'$gt' : stimeutc.timestamp,'$lt' : etimeutc.timestamp}}
srcid_list=db.wf_miniseed.find(query).distinct("source_id")
print("Number of distinct sources that will be processed this run=",len(srcid_list))
t0=time.time()
srcidbag = bag.from_sequence(srcid_list)
bagout = srcidbag.map(set_Ptime_by_event,db,model)
bagout.compute()
t=time.time()
print('done: elapsed time=',t-t0)Note this is set_Ptime_by_event:
from obspy import UTCDateTime
from obspy.taup import TauPyModel
from obspy.geodetics import locations2degrees
def set_Ptime_by_event(srcid,db,model):
print('working on data for source_id=',srcid)
phase_list=['P']
doc = db.source.find_one({"_id" : srcid})
if doc == None:
return [0,0]
#srcid=doc['_id']
srclat=doc['lat']
srclon=doc['lon']
srcdep=doc['depth']
srctime=doc['time']
print("epicenter: ",srclat,srclon)
# Use this form for first pass. Second if a lot of the work was done
# Had first run abort with a failure that provided no helpful diagnotics
#query = {"source_id" : srcid}
query = {"$and" : [
{"source_id" : {"$eq" : srcid}},
{"Ptime" : {"$exists" : False}}]}
n=db.wf_miniseed.count_documents(query)
print('Number of Ptimes this run will compute=',n)
if n == 0:
return [0,0]
wfcurs=db.wf_miniseed.find(query,no_cursor_timeout=True)
number_with_no_match=0
for wfdoc in wfcurs:
wfid = wfdoc['_id']
net=wfdoc['net']
sta=wfdoc['sta']
chan=wfdoc['chan']
if 'loc' in wfdoc:
loc = wfdoc['loc']
stadoc=db.get_seed_channel(net,sta,chan,loc,srctime)
else:
stadoc=db.get_seed_channel(net,sta,chan,time=srctime)
if stadoc == None:
number_with_no_match += 1
else:
stalat=stadoc['lat']
stalon=stadoc['lon']
staelev=stadoc['elev']
degdist=locations2degrees(srclat,srclon,stalat,stalon)
arr=model.get_travel_times(distance_in_degree=degdist,
source_depth_in_km=srcdep,
phase_list=phase_list)
# get_travel_times returns an empty list (not well document in obspy) if it cannot
# compute a P phase. That happens for any source-receiver pair in the core shadow.
# We silently drop those with this conditional
if len(arr)>0:
atime=srctime+arr[0].time
db.wf_miniseed.update_one({'_id' : wfid},
{'$set' : {'Ptime' : atime,'P_travel_time' : arr[0].time,'epicentral_distance' : degdist}})
return [number_with_no_match,n]This is anything but a lightweight processing job as the 2012 wf_miniseed collection has 3.5 M documents. As you can see this is a nearly pure database process. The driver assembles a list of source_id keys it uses to define the bag for processing with (in this case) dask. (Note I haven't tried this with spark - prehaps it would be a useful test but I'd need some hand holding to get the incantations right) The function queries wf_miniseed by source_id and then (works through every document matching the query for that source_id in wf_miniseed), computes the desired time, and then updates the appropriate wf_miniseed document adding "Ptime" and "epicentral_distance". Note the one-at-at-time transaction here could probably be done faster if I used update_many, but that is a performance issue that is unllikely related to crashing mongodb.
I am a bit suspicious that my use of the "no_cursor_timeout=True" option here is what is crashing MongoDB. I can imagine it would cause the server to not release resources. I'm going to test that hypothesis next by setting the option false, but I can't do it immediately. The machine is remote and I cannot find out to restart mongodb from the jupyter-notebook terminal. We probably need a wiki entry on how to do that I can see that happening to our users down the road.
A competing hypothesis for crashing is that stdout messages are causing an issue. What I've found mysterious is that a serial version of this same task produces voluminous output that led me to initiate this issue page. I have found the parallel version is totally silent, at least running under jupyter-notebook. The log should be millions of lines (if the job ran to completion) that is mysterious - the message noted at the top of this post is posted for a significant fraction of the 3.5 million documents.
We need to fix a nonfatal but problematic issue with two methods of Database called
get_seed_siteandget_seed_channel. Both have diagnostic print messages to warn about a problem with a match that yield more than one document. The solution needed for the short term is to add a verbose option to the arg list to both methods that silence these messages by default. The condition is usually harmless when it happens and is created by metadata errors in the master database at IRIS - it also fairly common. That is why the default should be verbose=False.These two methods actually have a disconnect with the design of MsPASS for handling errors like this. Both conditions would be appropriate to log to data with an
ErrorSeverity.Complaint, but the api for these two function has no way to do that without throwing an exception. I don't think that is a good idea either, however, as this "exception" is not that exceptional. Throwing an exception could cause users unexpected aborts from the functions being excessively pedantic and slow down processing when an error handler was included. For that reason I'm not totally sure how we should fix this short of creating a completely different function or method to accomplish the same thing but which handles this more cleanly. That may require some discussion in the group.For now let's add the verbose option and put this on the longer term development agenda.