search

MetPX / sarracenia

https://MetPX.github.io/sarracenia
GNU General Public License v2.0
45 stars 22 forks source link

Circular reference ERROR detected while running sr_sarra to fetch PODAAC files #344

Open iMacadan opened 4 years ago

iMacadan commented 4 years ago 
sarra@data-server1:~/.config/sarra$ sr_sarra debug foreground testing_get-podaac-from-data-server-science
​
2020-06-26 16:07:09,360 [INFO] file_log downloaded to: /home/apps/sarra/public_data/test_data/20200626/PODAAC/20200625233000-OSPO-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.61-v02.0-fv01.0.nc
2020-06-26 16:07:09,360 [DEBUG] sr_sarra __on_post__
2020-06-26 16:07:09,360 [INFO] post_log notice=20200626010506.346583128 http://data-server1.science.example.com/ 20200626/PODAAC/20200625233000-OSPO-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.61-v02.0-fv01.0.nc headers={'oldname': {...}, 'atime': '20200626010505.824895144', 'to_clusters': 'DATASRV.SCIENCE,DATASRV.METEO,DATASRV1.METEO,DATASRV1.SCIENCE,DATASRV.STAGE,DATASRV.DEV', 'mode': '444', 'sum': 'd,bc20bde554e57d83e85470b22ff71a56', 'parts': '1,4735167,1,0,0', 'from_cluster': 'SCIENCE', 'mtime': '20200626010020', 'source': 'PODAAC'}
2020-06-26 16:07:09,361 [ERROR] sr_sarra/run going badly with Circular reference detected, sleeping for 0.010.
2020-06-26 16:07:09,361 [DEBUG] Exception details: 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 1759, in run
    ok = self.process_message()
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 942, in process_message
    ok = self.__do_tasks__()
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 421, in __do_tasks__
    if not plugin(self): return False
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 1571, in doit_download
    ok = self.__on_post__()
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 681, in __on_post__
    ok = self.msg.post(self)
  File "/usr/lib/python3/dist-packages/sarra/sr_message.py", line 700, in post
    self.headers[h] = json.dumps( self.headers[h] )
  File "/usr/lib/python3.5/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected
2020-06-26 16:07:09,363 [DEBUG] Encoding msg to json: message={'body': '20200626010506.346583128 http://data-server2.science.example.com /PODAAC/177/20200625233000-OSPO-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.61-v02.0-fv01.0.nc', 'isRetry': False, 'delivery_info': {'message_count': 552, 'exchange': 'xs_PODAAC', 'delivery_tag': 2, 'redelivered': False, 'routing_key': 'v02.post.PODAAC.177'}, 'properties': {'content_encoding': 'UTF-8', 'application_headers': {'oldname': {...}, 'atime': '20200626010505.824895144', 'to_clusters': 'DATASRV.SCIENCE,DATASRV.METEO,DATASRV1.METEO,DATASRV1.SCIENCE,DATASRV.STAGE,DDSR.DEV', 'mode': '444', 'sum': 'd,bc20bde554e57d83e85470b22ff71a56', 'parts': '1,4735167,1,0,0', 'from_cluster': 'SCIENCE', 'mtime': '20200626010020', 'source': 'PODAAC'}, 'content_type': 'text/plain'}, 'channel': <amqp.channel.Channel object at 0x7f1c8a7b3390>}
2020-06-26 16:07:09,363 [ERROR] failed to serialize message to JSON: 20200626010506.346583128 http://data-server2.science.example.com /PODAAC/177/20200625233000-OSPO-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.61-v02.0-fv01.0.nc
2020-06-26 16:07:09,364 [DEBUG] Exception details:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 1759, in run
    ok = self.process_message()
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 942, in process_message
    ok = self.__do_tasks__()
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 421, in __do_tasks__
    if not plugin(self): return False
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 1571, in doit_download
    ok = self.__on_post__()
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 681, in __on_post__
    ok = self.msg.post(self)
  File "/usr/lib/python3/dist-packages/sarra/sr_message.py", line 700, in post
    self.headers[h] = json.dumps( self.headers[h] )
  File "/usr/lib/python3.5/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sarra/sr_retry.py", line 310, in msg_append_to_file
    line = self.msgToJSON(message,done)
  File "/usr/lib/python3/dist-packages/sarra/sr_retry.py", line 185, in msgToJSON
    return json.dumps([topic, headers, notice], sort_keys=True) + '\n'
  File "/usr/lib/python3.5/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected
iMacadan commented 4 years ago

Note:
We detect this issue when looking at the logs.
Otherwise, files transfer seems to work fine...

iMacadan commented 4 years ago

Another type of error showed up in logs 

2020-06-29 13:20:07,193 [ERROR] sr_sarra/run going badly with can only concatenate tuple (not "str") to tuple, sleeping for 0.010.
2020-06-29 13:20:07,193 [DEBUG] Exception details: 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 1759, in run
    ok = self.process_message()
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 942, in process_message
    ok = self.__do_tasks__()
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 421, in __do_tasks__
    if not plugin(self): return False
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 1507, in doit_download
    if not self.__on_file__():
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 636, in __on_file__
    self.__plugin_backward_compat_setup__()
  File "/usr/lib/python3/dist-packages/sarra/sr_subscribe.py", line 823, in __plugin_backward_compat_setup__
    self.pbc_new_url     = urllib.parse.urlparse(self.msg.new_baseurl + '/' + self.msg.new_relpath)
TypeError: can only concatenate tuple (not "str") to tuple
2020-06-29 13:20:07,196 [DEBUG] Encoding msg to json: message={'channel': <amqp.channel.Channel object at 0x7f034959e5f8>, 'body': '20200629132006.312653542 http://data-server2.science.example.com /PODAAC/181/20200629115000-O
SPO-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.61-v02.0-fv01.0.nc', 'delivery_info': {'routing_key': 'v02.post.PODAAC.181', 'redelivered': False, 'exchange': 'xs_PODAAC', 'delivery_tag': 2, 'message_count': 0}, 'i
sRetry': False, 'properties': {'content_type': 'text/plain', 'application_headers': {'parts': '1,1072765,1,0,0', 'mtime': '20200629131921', 'from_cluster': 'SCIENCE', 'to_clusters': 'DATASRV.SCIENCE,DATASRV.METEO,DATASRV1.METEO,DATASRV1.SCIENCE,DATASRV.STAGE,DDSR.DEV', 'sum': 'd,2a8a624accd9be4cdbe7ced3c4e386ba', 'source': 'PODAAC', 'atime': '20200629132005.720555782', 'oldname': '/PODAAC/181/.20200629115000-OSPO-L3U_GHRSST-SSTsubskin-VIIRS_
NPP-ACSPO_V2.61-v02.0-fv01.0.nc.6okc0U', 'mode': '444'}, 'content_encoding': 'UTF-8'}}