search

dmwm / DDM

Dynamic Data Management - Cache release and auto-replication of hot data
1 stars 9 forks source link

Crab popularity collector returns meaningless error message if MV refresh fails #83

Open nikmagini opened 9 years ago

nikmagini commented 9 years ago

When an MV refresh procedure fails, the CRAB popdb collector script popdbRefresh.py returns an obscure error message [*].

The reason is that multiprocessing.Pool used by popdbRefresh.py to run the refresh procedures in parallel requires that all returns - including errors - are pickleable.

The Oracle exceptions in the _refresh_SingleMV functions should be properly caught and handled to avoid being propagated upwards and triggering the UnpickingError from multiprocessing.Pool

[*]

INFO:data popularity:got connection for _refresh_SingleMV table MV_block_stat0, mode F INFO:data popularity:got connection for _refresh_SingleMV table MV_DS_Files, mode F INFO:data popularity:got connection for _refresh_SingleMV table MV_USER_USERID, mode F Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(_self.args, *_self.__kwargs) File "/usr/lib64/python2.6/multiprocessing/pool.py", line 259, in _handle_results task = get()

UnpicklingError: NEWOBJ class argument has NULL tp_new