Issue in site_test on theta

[yongyanrao]

Site test on theta for 1.1.0a1: commit 3687404abf2eb00d3baa5076c70813f5d7bf8646 (HEAD -> master, tag: 1.1.0a1, origin/master, origin/HEAD)

Under conda env (miniconda-3/latest//gpfs/mira-home/yrao2/Documents/parsl/parsl_3687404a.py3.7), executed pytest parsl/tests/site_tests/test_site.py -k "not cleannet" --config local

After it began running, it seems hanging there forever without returning. But in fact, it has been killed by the scheduler after 10 minutes scheduled limit. Needed to ctrl+c to end it. In parsl.log, it says

2021-03-05 22:41:45.092 parsl.process_loggers:31 [ERROR]  Exceptional ending for _queue_management_worker on thread HTEX-Queue-Management-Thread
Traceback (most recent call last):
  File "/gpfs/mira-home/yrao2/Documents/parsl/parsl/process_loggers.py", line 27, in wrapped
    r = func(*args, **kwargs)
  File "/gpfs/mira-home/yrao2/Documents/parsl/parsl/executors/high_throughput/executor.py", line 402, in _queue_management_worker
    result = deserialize(msg['result'])
  File "/gpfs/mira-home/yrao2/Documents/parsl/parsl/serialize/facade.py", line 121, in deserialize
    result = self.methods_for_code[header].deserialize(payload)
  File "/gpfs/mira-home/yrao2/Documents/parsl/parsl/serialize/concretes.py", line 28, in deserialize
    data = pickle.loads(chomped)
TypeError: <lambda>() takes 6 positional arguments but 7 were given

[yongyanrao]

It turns out it is due to pickle version, as Python 3.9.1 was used before. If we used Python 3.8.8, it should work.

[benclifford]

@yongyanrao do you mean that you used different python versions within the same run? (a misconfiguration)

Or that this test did not work with python 3.9.1 at all? (a problem with parsl)

[yongyanrao]

@benclifford I think it is the second scenario. I needed to install a brand new conda environment based on Python 3.8.8 to make it work.