search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
27.69k stars 4.09k forks source link

[Bug]: Bus error (core dumped) #8974

Open SpaceHunterInf opened 1 day ago

SpaceHunterInf commented 1 day ago

Your current environment

The output of `python collect_env.py` ```text Collecting environment information... PyTorch version: 2.4.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.5 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.26.4 Libc version: glibc-2.35 Python version: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.3.107 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: Quadro RTX 8000 Nvidia driver version: 545.23.08 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.7 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.7 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.7 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.7 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.7 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.7 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.7 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz CPU family: 6 Model: 85 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 8 Stepping: 7 BogoMIPS: 4589.32 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single intel_ppin ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves pku ospke md_clear flush_l1d arch_capabilities Hypervisor vendor: Xen Virtualisation type: full L1d cache: 256 KiB (8 instances) L1i cache: 256 KiB (8 instances) L2 cache: 8 MiB (8 instances) L3 cache: 176 MiB (8 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-7 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Versions of relevant libraries: [pip3] numpy==1.23.5 [pip3] nvidia-cublas-cu11==11.10.3.66 [pip3] nvidia-cublas-cu12==12.1.3.1 [pip3] nvidia-cuda-cupti-cu11==11.7.101 [pip3] nvidia-cuda-cupti-cu12==12.1.105 [pip3] nvidia-cuda-nvrtc-cu11==11.7.99 [pip3] nvidia-cuda-nvrtc-cu12==12.1.105 [pip3] nvidia-cuda-runtime-cu11==11.7.99 [pip3] nvidia-cuda-runtime-cu12==12.1.105 [pip3] nvidia-cudnn-cu11==8.5.0.96 [pip3] nvidia-cudnn-cu12==9.1.0.70 [pip3] nvidia-cufft-cu11==10.9.0.58 [pip3] nvidia-cufft-cu12==11.0.2.54 [pip3] nvidia-curand-cu11==10.2.10.91 [pip3] nvidia-curand-cu12==10.3.2.106 [pip3] nvidia-cusolver-cu11==11.4.0.1 [pip3] nvidia-cusolver-cu12==11.4.5.107 [pip3] nvidia-cusparse-cu11==11.7.4.91 [pip3] nvidia-cusparse-cu12==12.1.0.106 [pip3] nvidia-ml-py==12.560.30 [pip3] nvidia-ml-py3==7.352.0 [pip3] nvidia-nccl-cu11==2.14.3 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] nvidia-nvjitlink-cu12==12.3.52 [pip3] nvidia-nvtx-cu11==11.7.91 [pip3] nvidia-nvtx-cu12==12.1.105 [pip3] pyzmq==26.2.0 [pip3] torch==2.4.0 [pip3] torch-fidelity==0.3.0 [pip3] torchinfo==1.8.0 [pip3] torchmetrics==1.4.0.post0 [pip3] torchvision==0.19.0 [pip3] transformers==4.45.1 [pip3] triton==3.0.0 [conda] cuda-version 12.6 3 nvidia [conda] nccl 2.21.5.1 ha515578_0 ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.5.4@4db5176d9758b720b05460c50ace3c01026eb158 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X 0-7 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ```

Model Input Dumps

No response

🐛 Describe the bug

Greetings, I was using vllm 0.5.2 just fine, but I want to do some constrained decoding and therefore upgraded to 0.6.2. Then I always encouter this Bus error when I attempt to inference.

I've tried to delete and re-create conda environment, downgrade to 0.5.4 uninstall/reinstall nccl from source/apt/conda multiple times (according to #3916) but non of this helped. However, my other machine works perfectly fine. My code is very standard

sampling_params = SamplingParams(temperature=0, top_p=1, max_tokens=20)
data =  [out]
outputs = llm.generate(data, sampling_params, guided_options_request=dict(guided_choice=["agree", "disagree"
    ]))
output_batch = [output.outputs[0].text for output in outputs]

Could someone help me out?

Much appreciated

bt from gdp core file

Core was generated by `python3 vllm_mpd.py'.
Program terminated with signal SIGBUS, Bus error.
#0  0x00007f7a3cc9ab8f in ?? ()
[Current thread is 1 (LWP 2829379)]
(gdb) bt
#0  0x00007f7a3cc9ab8f in ?? ()
#1  0x00007f7a58845064 in ?? ()
#2  0x0000556600000001 in ?? ()
#3  0x0000000000000009 in ?? ()
#4  0x00005566001e7999 in ?? ()
#5  0x00005566001e7999 in ?? ()
#6  0x7fffffffffffffff in ?? ()
#7  0x00007fffafe13320 in ?? ()
#8  0x00005565fffcdaf4 in ?? ()
#9  0x00005566020f7920 in ?? ()
#10 0x00007f7b1dd94ef0 in ?? ()
#11 0x00007f7a360405e8 in ?? ()
#12 0x00007f7b1e61ace0 in ?? ()
#13 0x00007f7a36040440 in ?? ()
#14 0x70941bcbe0b11000 in ?? ()
#15 0x00007f7a36040440 in ?? ()
#16 0x0000000000000001 in ?? ()
#17 0x00007fffafe1334c in ?? ()
#18 0x0000556608f286a8 in ?? ()
#19 0x0000556609238148 in ?? ()
#20 0x0000000000000000 in ?? ()

Before submitting a new issue...

DarkLight1337 commented 20 hours ago

Which model are you using?

@youkaichao any thoughts on this?

youkaichao commented 20 hours ago

the information is not very helpful. maybe follow https://docs.vllm.ai/en/latest/getting_started/debugging.html and provide more information?

SpaceHunterInf commented 11 hours ago

Hi, Thanks for your reply. I've followed the instructions from the debugging page. I write a minimal script, vllm_debug.py

from vllm import LLM, SamplingParams

if __name__ == '__main__':
    llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct", dtype='float32')
    sampling_params = SamplingParams(temperature=0, top_p=1, max_tokens=20)
    data = ['Do you agree that the sun is larger than the earth?', 'Do you agree the moon is larger than the earth?']
    outputs = llm.generate(data, sampling_params, guided_options_request=dict(guided_choice=["agree", "disagree"]))
    output_batch = [output.outputs[0].text for output in outputs]
    print(outputs[0].outputs[0].text)

I've run it with all four flags on

VLLM_LOGGING_LEVEL=DEBUG CUDA_LAUNCH_BLOCKING=1 NCCL_DEBUG=TRACE VLLM_TRACE_FUNCTION=1 python vllm_debug.py

But no additional error message is given except for the 'Bus Error'.

I've also run the test.py file to check GPU communications, it returns the following error

PyTorch NCCL is successful!
PyTorch GLOO is successful!
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/xz479/conformity/llm-conformity/vllm_nccl_debug.py", line 34, in <module>
[rank0]:     pynccl.all_reduce(data, stream=s)
[rank0]:   File "/home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl.py", line 113, in all_reduce
[rank0]:     assert tensor.device == self.device, (
[rank0]: AttributeError: 'PyNcclCommunicator' object has no attribute 'device'
[rank0]:[W1001 13:38:21.097554047 ProcessGroupNCCL.cpp:1168] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
dev-gpu-xz479:3044044:3044084 [0] NCCL INFO [Service thread] Connection closed by localRank 0
dev-gpu-xz479:3044044:3044114 [0] NCCL INFO comm 0x88a34f0 rank 0 nranks 1 cudaDev 0 busId 50 - Abort COMPLETE
E1001 13:38:22.706371 140180943778880 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 3044044) of binary: /home/xz479/miniconda3/envs/conformity/bin/python
Traceback (most recent call last):
  File "/home/xz479/miniconda3/envs/conformity/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
vllm_nccl_debug.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-10-01_13:38:22
  host      : dev-gpu-xz479.cl.cam.ac.uk
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 3044044)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

I've followed #8420, but it's not really the case. As I created a new conda environment and directly run the command 'pip install vllm'. I am downgrading to python 3.9.19 with vllm 0.5.4, which works perfectly fine on my other machine. I really wonder why. Everything worked before, and my whole machine collapses after I upgrade my vllm to 0.6.2, and it keeps saying Bus Error even after I, reinstalled nccl, create a separate new conda environment, and downgrade packages.

Thank you very much for your support.

Here is the bt from the latest core dump, it shows something different.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--
Core was generated by `python vllm_debug.py'.
Program terminated with signal SIGBUS, Bus error.
#0  0x00007fb63b224c37 in walTryBeginRead () from /home/xz479/miniconda3/envs/conformity/lib/python3.9/lib-dynload/../../libsqlite3.so.0
[Current thread is 1 (Thread 0x7fb7026f8440 (LWP 3046686))]
(gdb) 
(gdb) bt
#0  0x00007fb63b224c37 in walTryBeginRead () from /home/xz479/miniconda3/envs/conformity/lib/python3.9/lib-dynload/../../libsqlite3.so.0
#1  0x00007fb63b249cc0 in sqlite3PagerSharedLock () from /home/xz479/miniconda3/envs/conformity/lib/python3.9/lib-dynload/../../libsqlite3.so.0
#2  0x00007fb63b24a7d8 in btreeBeginTrans () from /home/xz479/miniconda3/envs/conformity/lib/python3.9/lib-dynload/../../libsqlite3.so.0
#3  0x00007fb63b28b6c0 in sqlite3VdbeExec () from /home/xz479/miniconda3/envs/conformity/lib/python3.9/lib-dynload/../../libsqlite3.so.0
#4  0x00007fb63b2911af in sqlite3_step () from /home/xz479/miniconda3/envs/conformity/lib/python3.9/lib-dynload/../../libsqlite3.so.0
#5  0x00007fb63b3246f6 in pysqlite_step (connection=<optimised out>, statement=0x842eb38) at /usr/local/src/conda/python-3.9.19/Modules/_sqlite/util.c:37
#6  _pysqlite_query_execute (self=<optimised out>, multiple=0, args=<optimised out>) at /usr/local/src/conda/python-3.9.19/Modules/_sqlite/cursor.c:526
#7  0x00000000005073c0 in cfunction_call (func=0x7fb5f8166b80, args=<optimised out>, kwargs=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/methodobject.c:552
#8  0x000000000050597a in _PyObject_Call (tstate=0x14685e0, callable=0x7fb5f8166b80, args=0x7fb606570300, kwargs=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:281
#9  0x00007fb63b3240d1 in pysqlite_connection_execute (self=<optimised out>, args=0x7fb606570300)
    at /usr/local/src/conda/python-3.9.19/Modules/_sqlite/connection.c:1334
#10 0x00000000005073c0 in cfunction_call (func=0x7fb5f8166a40, args=<optimised out>, kwargs=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/methodobject.c:552
#11 0x00000000004f065c in _PyObject_MakeTpCall (tstate=0x14685e0, callable=0x7fb5f8166a40, args=<optimised out>, nargs=<optimised out>, keywords=0x0)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:191
#12 0x000000000042a763 in call_function (kwnames=0x0, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5074
#13 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x82ccef0, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3489
#14 0x00000000004e69da in _PyEval_EvalFrame (throwflag=0, f=0x82ccef0, tstate=0x14685e0)
--Type <RET> for more, q to quit, c to continue without paging--
   .h:40
#15 _PyEval_EvalCode (tstate=<optimised out>, _co=<optimised out>, globals=<optimised out>, locals=<optimised out>, args=<optimised out>, 
    argcount=<optimised out>, kwnames=0x7fb63b3b1318, kwargs=0x7fb5f82bdd90, kwcount=<optimised out>, kwstep=1, defs=0x7fb63b356698, defcount=<optimised out>, 
    kwdefs=0x0, closure=0x0, name=0x7fb7026c5df0, qualname=0x7fb63b3500f0) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:4329
#16 0x000000000050509d in _PyFunction_Vectorcall (kwnames=<optimised out>, nargsf=<optimised out>, stack=0x7fb5f82bdd80, func=0x7fb63b3661f0)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:396
#17 _PyObject_VectorcallTstate (kwnames=<optimised out>, nargsf=<optimised out>, args=0x7fb5f82bdd80, callable=0x7fb63b3661f0, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#18 method_vectorcall (method=method@entry=0x7fb5f8165ec0, args=0x7fb5f82bdd88, nargsf=<optimised out>, kwnames=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/classobject.c:53
#19 0x00000000004ef4c5 in _PyObject_VectorcallTstate (tstate=0x14685e0, callable=0x7fb5f8165ec0, args=<optimised out>, nargsf=<optimised out>, 
    kwnames=<optimised out>) at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#20 0x000000000042a6b7 in call_function (kwnames=0x7fb63b3b1300, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5074
#21 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x7fb5f82bdbe0, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3537
#22 0x00000000004e69da in _PyEval_EvalFrame (throwflag=0, f=0x7fb5f82bdbe0, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#23 _PyEval_EvalCode (tstate=<optimised out>, _co=<optimised out>, globals=<optimised out>, locals=<optimised out>, args=<optimised out>, 
    argcount=<optimised out>, kwnames=0x0, kwargs=0x7fb5f8189b08, kwcount=<optimised out>, kwstep=1, defs=0x0, defcount=<optimised out>, kwdefs=0x0, 
    closure=0x7fb60b8d0980, name=0x7fb701d5c030, qualname=0x7fb63b3b0210) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:4329
#24 0x000000000050502c in _PyFunction_Vectorcall (kwnames=<optimised out>, nargsf=<optimised out>, stack=0x7fb5f8189af0, func=0x7fb60ade5dc0)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:396
#25 _PyObject_VectorcallTstate (kwnames=<optimised out>, nargsf=<optimised out>, args=0x7fb5f8189af0, callable=0x7fb60ade5dc0, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#26 method_vectorcall (method=method@entry=0x7fb5f8165e40, args=0x7fb5f8189af8, nargsf=<optimised out>, kwnames=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/classobject.c:53
#27 0x00000000004ef4c5 in _PyObject_VectorcallTstate (tstate=0x14685e0, callable=0x7fb5f8165e40, args=<optimised out>, nargsf=<optimised out>, 
    kwnames=<optimised out>) at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#28 0x000000000042a763 in call_function (kwnames=0x0, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5074
#29 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x7fb5f8189950, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3489
#30 0x00000000004e69da in _PyEval_EvalFrame (throwflag=0, f=0x7fb5f8189950, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#31 _PyEval_EvalCode (tstate=<optimised out>, _co=<optimised out>, globals=<optimised out>, locals=<optimised out>, args=<optimised out>, 
    argcount=<optimised out>, kwnames=0x0, kwargs=0x7ffe309558d8, kwcount=<optimised out>, kwstep=1, defs=0x0, defcount=<optimised out>, kwdefs=0x0, 
    closure=0x7fb64d300190, name=0x7fb7026b5530, qualname=0x7fb64d31bda0) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:4329
#32 0x00000000004efece in _PyFunction_Vectorcall (kwnames=0x0, nargsf=<optimised out>, stack=<optimised out>, func=0x7fb63b38d550)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:396
#33 _PyObject_FastCallDictTstate (tstate=0x14685e0, callable=0x7fb63b38d550, args=<optimised out>, nargsf=<optimised out>, kwargs=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:118
#34 0x00000000005027bf in _PyObject_Call_Prepend (kwargs=0x0, args=0x7fb62271f6c0, obj=<optimised out>, callable=0x7fb63b38d550, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:489
#35 slot_tp_init (self=<optimised out>, args=0x7fb62271f6c0, kwds=0x0) at /usr/local/src/conda/python-3.9.19/Objects/typeobject.c:6971
#36 0x00000000004f0673 in type_call (kwds=0x0, args=0x7fb62271f6c0, type=<optimised out>) at /usr/local/src/conda/python-3.9.19/Objects/typeobject.c:1028
#37 _PyObject_MakeTpCall (tstate=0x14685e0, callable=0x8f555f0, args=<optimised out>, nargs=<optimised out>, keywords=0x0)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:191
#38 0x000000000042b593 in call_function (kwnames=0x0, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5074
#39 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x7fb5f8168580, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3520
#40 0x00000000004f80b3 in _PyEval_EvalFrame (throwflag=0, f=0x7fb5f8168580, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#41 function_code_fastcall (tstate=0x14685e0, co=<optimised out>, args=<optimised out>, nargs=<optimised out>, globals=0x7fb64d30ed40)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:330
#42 0x00000000004ef4c5 in _PyObject_VectorcallTstate (tstate=0x14685e0, callable=0x7fb60ade7430, args=<optimised out>, nargsf=<optimised out>, 
--Type <RET> for more, q to quit, c to continue without paging--
    kwnames=<optimised out>) at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#43 0x000000000042b593 in call_function (kwnames=0x0, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5074
#44 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x7fb5f8189780, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3520
#45 0x00000000004f80b3 in _PyEval_EvalFrame (throwflag=0, f=0x7fb5f8189780, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#46 function_code_fastcall (tstate=0x14685e0, co=<optimised out>, args=<optimised out>, nargs=<optimised out>, globals=0x7fb64d30ed40)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:330
#47 0x00000000004ef4c5 in _PyObject_VectorcallTstate (tstate=0x14685e0, callable=0x7fb64d3ed9d0, args=<optimised out>, nargsf=<optimised out>, 
    kwnames=<optimised out>) at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#48 0x000000000042b593 in call_function (kwnames=0x0, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5074
#49 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x7fb5f8168040, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3520
#50 0x00000000004f80b3 in _PyEval_EvalFrame (throwflag=0, f=0x7fb5f8168040, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#51 function_code_fastcall (tstate=0x14685e0, co=<optimised out>, args=<optimised out>, nargs=<optimised out>, globals=0x7fb64df3c880)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:330
#52 0x00000000004ef4c5 in _PyObject_VectorcallTstate (tstate=0x14685e0, callable=0x7fb60ade74c0, args=<optimised out>, nargsf=<optimised out>, 
    kwnames=<optimised out>) at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#53 0x000000000042b593 in call_function (kwnames=0x0, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5074
#54 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x7fb5f81895b0, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3520
#55 0x00000000004e69da in _PyEval_EvalFrame (throwflag=0, f=0x7fb5f81895b0, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#56 _PyEval_EvalCode (tstate=<optimised out>, _co=<optimised out>, globals=<optimised out>, locals=<optimised out>, args=<optimised out>, 
    argcount=<optimised out>, kwnames=0x0, kwargs=0x2d2e78d0, kwcount=<optimised out>, kwstep=1, defs=0x7fb624cc9328, defcount=<optimised out>, kwdefs=0x0, 
    closure=0x0, name=0x7fb64df3d350, qualname=0x7fb64df3d4e0) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:4329
#57 0x00000000004f7de4 in _PyFunction_Vectorcall (func=<optimised out>, stack=<optimised out>, nargsf=<optimised out>, kwnames=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:396
#58 0x00000000004ef4c5 in _PyObject_VectorcallTstate (tstate=0x14685e0, callable=0x7fb60ade7dc0, args=<optimised out>, nargsf=<optimised out>, 
    kwnames=<optimised out>) at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#59 0x000000000042b81d in call_function (kwnames=0x0, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5074
#60 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x2d2e76f0, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3506
#61 0x00000000004e69da in _PyEval_EvalFrame (throwflag=0, f=0x2d2e76f0, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#62 _PyEval_EvalCode (tstate=<optimised out>, _co=<optimised out>, globals=<optimised out>, locals=<optimised out>, args=<optimised out>, 
    argcount=<optimised out>, kwnames=0x7fb624cb5aa8, kwargs=0x27965af0, kwcount=<optimised out>, kwstep=1, defs=0x7fb624cb2b08, defcount=<optimised out>, 
    kwdefs=0x0, closure=0x0, name=0x7fb64df3d210, qualname=0x7fb64df3d3a0) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:4329
#63 0x000000000050509d in _PyFunction_Vectorcall (kwnames=<optimised out>, nargsf=<optimised out>, stack=0x27965ae8, func=0x7fb60ade7ca0)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:396
#64 _PyObject_VectorcallTstate (kwnames=<optimised out>, nargsf=<optimised out>, args=0x27965ae8, callable=0x7fb60ade7ca0, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#65 method_vectorcall (method=method@entry=0x7fb5f8165d80, args=0x27965af0, nargsf=<optimised out>, kwnames=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/classobject.c:53
#66 0x00000000004ef4c5 in _PyObject_VectorcallTstate (tstate=0x14685e0, callable=0x7fb5f8165d80, args=<optimised out>, nargsf=<optimised out>, 
    kwnames=<optimised out>) at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#67 0x000000000042a6b7 in call_function (kwnames=0x7fb624cb5a90, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5074
#68 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x27965930, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3537
#69 0x00000000004e69da in _PyEval_EvalFrame (throwflag=0, f=0x27965930, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#70 _PyEval_EvalCode (tstate=<optimised out>, _co=<optimised out>, globals=<optimised out>, locals=<optimised out>, args=<optimised out>, 
    argcount=<optimised out>, kwnames=0x7fb5f82df958, kwargs=0x7fb701dd8d40, kwcount=<optimised out>, kwstep=1, defs=0x7fb64e2377d8, defcount=<optimised out>, 
--Type <RET> for more, q to quit, c to continue without paging--
    kwdefs=0x0, closure=0x0, name=0x7fb701dd7df0, qualname=0x7fb64df3c130) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:4329
#71 0x00000000004f7de4 in _PyFunction_Vectorcall (func=<optimised out>, stack=<optimised out>, nargsf=<optimised out>, kwnames=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:396
#72 0x00000000005057e4 in PyVectorcall_Call (kwargs=<optimised out>, tuple=<optimised out>, callable=0x7fb60ade79d0)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:243
#73 _PyObject_Call (kwargs=<optimised out>, args=<optimised out>, callable=0x7fb60ade79d0, tstate=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:266
#74 PyObject_Call (callable=0x7fb60ade79d0, args=<optimised out>, kwargs=<optimised out>) at /usr/local/src/conda/python-3.9.19/Objects/call.c:293
#75 0x00000000004eba52 in do_call_core (kwdict=0x7fb5f8165080, callargs=0x7fb5f81b1340, func=0x7fb60ade79d0, tstate=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5125
#76 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x7fb5f82bb9a0, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3582
#77 0x00000000004e69da in _PyEval_EvalFrame (throwflag=0, f=0x7fb5f82bb9a0, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#78 _PyEval_EvalCode (tstate=<optimised out>, _co=<optimised out>, globals=<optimised out>, locals=<optimised out>, args=<optimised out>, 
    argcount=<optimised out>, kwnames=0x7fb70268c508, kwargs=0x14c5ac0, kwcount=<optimised out>, kwstep=1, defs=0x0, defcount=<optimised out>, kwdefs=0x0, 
    closure=0x7fb60ba21180, name=0x7fb701dd7df0, qualname=0x7fb64df3c130) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:4329
#79 0x000000000050509d in _PyFunction_Vectorcall (kwnames=<optimised out>, nargsf=<optimised out>, stack=0x14c5aa8, func=0x7fb60ade7a60)
    at /usr/local/src/conda/python-3.9.19/Objects/call.c:396
#80 _PyObject_VectorcallTstate (kwnames=<optimised out>, nargsf=<optimised out>, args=0x14c5aa8, callable=0x7fb60ade7a60, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#81 method_vectorcall (method=method@entry=0x7fb606572c80, args=0x14c5ab0, nargsf=<optimised out>, kwnames=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Objects/classobject.c:53
#82 0x00000000004ef4c5 in _PyObject_VectorcallTstate (tstate=0x14685e0, callable=0x7fb606572c80, args=<optimised out>, nargsf=<optimised out>, 
    kwnames=<optimised out>) at /usr/local/src/conda/python-3.9.19/Include/cpython/abstract.h:118
#83 0x000000000042a6b7 in call_function (kwnames=0x7fb70268c4f0, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=<optimised out>)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:5074
#84 _PyEval_EvalFrameDefault (tstate=<optimised out>, f=0x14c5940, throwflag=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3537
#85 0x00000000004e69da in _PyEval_EvalFrame (throwflag=0, f=0x14c5940, tstate=0x14685e0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#86 _PyEval_EvalCode (tstate=<optimised out>, _co=<optimised out>, globals=<optimised out>, locals=<optimised out>, args=<optimised out>, 
    argcount=<optimised out>, kwnames=0x0, kwargs=0x0, kwcount=<optimised out>, kwstep=2, defs=0x0, defcount=<optimised out>, kwdefs=0x0, closure=0x0, 
    name=0x0, qualname=0x0) at /usr/local/src/conda/python-3.9.19/Python/ceval.c:4329
#87 0x00000000004e6667 in _PyEval_EvalCodeWithName (_co=<optimised out>, globals=<optimised out>, locals=<optimised out>, args=<optimised out>, 
    argcount=<optimised out>, kwnames=<optimised out>, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:4361
#88 0x00000000004e6619 in PyEval_EvalCodeEx (_co=<optimised out>, globals=<optimised out>, locals=<optimised out>, args=<optimised out>, 
    argcount=<optimised out>, kws=<optimised out>, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:4377
#89 0x00000000005938eb in PyEval_EvalCode (co=co@entry=0x7fb701dc43a0, globals=globals@entry=0x7fb7023f8040, locals=locals@entry=0x7fb7023f8040)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:828
#90 0x00000000005c1157 in run_eval_code_obj (tstate=0x14685e0, co=0x7fb701dc43a0, globals=0x7fb7023f8040, locals=0x7fb7023f8040)
    at /usr/local/src/conda/python-3.9.19/Python/pythonrun.c:1221
#91 0x00000000005bd170 in run_mod (mod=<optimised out>, filename=<optimised out>, globals=0x7fb7023f8040, locals=0x7fb7023f8040, flags=<optimised out>, 
    arena=<optimised out>) at /usr/local/src/conda/python-3.9.19/Python/pythonrun.c:1242
#92 0x0000000000456423 in pyrun_file (fp=0x1467d50, filename=0x7fb7023bce30, start=<optimised out>, globals=0x7fb7023f8040, locals=0x7fb7023f8040, closeit=1, 
    flags=0x7ffe30956e98) at /usr/local/src/conda/python-3.9.19/Python/pythonrun.c:1140
#93 0x00000000005b6e52 in pyrun_simple_file (flags=0x7ffe30956e98, closeit=1, filename=0x7fb7023bce30, fp=0x1467d50)
    at /usr/local/src/conda/python-3.9.19/Python/pythonrun.c:450
#94 PyRun_SimpleFileExFlags (fp=0x1467d50, filename=<optimised out>, closeit=1, flags=0x7ffe30956e98)
    at /usr/local/src/conda/python-3.9.19/Python/pythonrun.c:483
#95 0x00000000005b43ce in pymain_run_file (cf=0x7ffe30956e98, config=0x1469150) at /usr/local/src/conda/python-3.9.19/Modules/main.c:379
#96 pymain_run_python (exitcode=0x7ffe30956e90) at /usr/local/src/conda/python-3.9.19/Modules/main.c:604
#97 Py_RunMain () at /usr/local/src/conda/python-3.9.19/Modules/main.c:683
#98 0x0000000000587999 in Py_BytesMain (argc=<optimised out>, argv=<optimised out>) at /usr/local/src/conda/python-3.9.19/Modules/main.c:1129
--Type <RET> for more, q to quit, c to continue without paging--
#99 0x00007fb702429d90 in __libc_start_call_main (main=main@entry=0x587950 <main>, argc=argc@entry=2, argv=argv@entry=0x7ffe309570c8)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#100 0x00007fb702429e40 in __libc_start_main_impl (main=0x587950 <main>, argc=2, argv=0x7ffe309570c8, init=<optimised out>, fini=<optimised out>, 
    rtld_fini=<optimised out>, stack_end=0x7ffe309570b8) at ../csu/libc-start.c:392
#101 0x000000000058784e in _start ()
SpaceHunterInf commented 11 hours ago

I am using model Llama-3-8B-Instruct, this machine is my dev machine with RTX Quadro 8000

youkaichao commented 6 hours ago

I've run it with all four flags on

VLLM_LOGGING_LEVEL=DEBUG CUDA_LAUNCH_BLOCKING=1 NCCL_DEBUG=TRACE VLLM_TRACE_FUNCTION=1 python vllm_debug.py

you need to paste the output log of this command

SpaceHunterInf commented 6 hours ago

Thank you for your help. However, the file is way too large (278MB). Here is the last 500 lines using tail command Again, thank you very much for your help, and patience. I am new to this. I am not sure if it contains the thing you need. Let me know if you need anything else.

2024-10-01 19:19:11.685462 Return from _verify_non_beam_search in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/sampling_params.py:276 to __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/sampling_params.py:194
2024-10-01 19:19:11.685492 Call to _verify_greedy_sampling in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/sampling_params.py:282 from __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/sampling_params.py:200
2024-10-01 19:19:11.685512 Return from _verify_greedy_sampling in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/sampling_params.py:283 to __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/sampling_params.py:200
2024-10-01 19:19:11.685541 Return from __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/sampling_params.py:202 to <module> in /home/xz479/conformity/llm-conformity/vllm_debug.py:5
2024-10-01 19:19:11.685584 Call to inner in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/utils.py:879 from <module> in /home/xz479/conformity/llm-conformity/vllm_debug.py:7
2024-10-01 19:19:11.685611 Call to <lambda> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:255 from inner in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/utils.py:881
2024-10-01 19:19:11.685635 Return from <lambda> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:255 to inner in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/utils.py:881
2024-10-01 19:19:11.685664 Call to generate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:253 from inner in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/utils.py:895
2024-10-01 19:19:11.685737 Call to _check_generic in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2872 from __getitem__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/typing.py:838
2024-10-01 19:19:11.685765 Return from _check_generic in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2884 to __getitem__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/typing.py:838
2024-10-01 19:19:11.685817 Call to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2989 from __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/typing.py:746
2024-10-01 19:19:11.685844 Call to _has_generic_or_protocol_as_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2954 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3001
2024-10-01 19:19:11.685871 Return from _has_generic_or_protocol_as_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2970 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3001
2024-10-01 19:19:11.685898 Call to _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2976 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3008
2024-10-01 19:19:11.685920 Call to get_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:1364 from _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2977
2024-10-01 19:19:11.685943 Return from get_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:1383 to _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2977
2024-10-01 19:19:11.685964 Return from _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2978 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3008
2024-10-01 19:19:11.685987 Call to _should_collect_from_parameters in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:159 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3023
2024-10-01 19:19:11.686008 Return from _should_collect_from_parameters in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:160 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3023
2024-10-01 19:19:11.686037 Call to <listcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3024 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3024
2024-10-01 19:19:11.686058 Return from <listcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3024 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3024
2024-10-01 19:19:11.686082 Return from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3025 to __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/typing.py:746
2024-10-01 19:19:11.686203 Call to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2989 from __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/typing.py:746
2024-10-01 19:19:11.686229 Call to _has_generic_or_protocol_as_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2954 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3001
2024-10-01 19:19:11.686252 Return from _has_generic_or_protocol_as_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2970 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3001
2024-10-01 19:19:11.686284 Call to _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2976 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3008
2024-10-01 19:19:11.686303 Call to get_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:1364 from _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2977
2024-10-01 19:19:11.686326 Return from get_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:1386 to _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2977
2024-10-01 19:19:11.686344 Return from _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2978 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3008
2024-10-01 19:19:11.686365 Call to _should_collect_from_parameters in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:159 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3023
2024-10-01 19:19:11.686383 Return from _should_collect_from_parameters in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:160 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3023
2024-10-01 19:19:11.686403 Call to _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2976 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3008
2024-10-01 19:19:11.686422 Call to get_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:1364 from _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2977
2024-10-01 19:19:11.686445 Return from get_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:1386 to _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2977
2024-10-01 19:19:11.686469 Return from _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2978 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3008
2024-10-01 19:19:11.686489 Call to _should_collect_from_parameters in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:159 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3023
2024-10-01 19:19:11.686508 Return from _should_collect_from_parameters in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:160 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3023
2024-10-01 19:19:11.686528 Call to _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2976 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3008
2024-10-01 19:19:11.686546 Call to get_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:1364 from _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2977
2024-10-01 19:19:11.686566 Return from get_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:1386 to _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2977
2024-10-01 19:19:11.686585 Return from _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2978 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3008
2024-10-01 19:19:11.686604 Call to _should_collect_from_parameters in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:159 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3023
2024-10-01 19:19:11.686623 Return from _should_collect_from_parameters in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:160 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3023
2024-10-01 19:19:11.686643 Call to _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2976 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3008
2024-10-01 19:19:11.686661 Call to get_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:1364 from _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2977
2024-10-01 19:19:11.686681 Return from get_origin in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:1383 to _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2977
2024-10-01 19:19:11.686699 Return from _is_unpacked_typevartuple in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:2978 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3008
2024-10-01 19:19:11.686719 Call to _should_collect_from_parameters in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:159 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3023
2024-10-01 19:19:11.686737 Return from _should_collect_from_parameters in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:160 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3023
2024-10-01 19:19:11.686760 Call to <listcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3024 from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3024
2024-10-01 19:19:11.686780 Return from <listcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3024 to _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3024
2024-10-01 19:19:11.686800 Return from _collect_type_vars in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/typing_extensions.py:3025 to __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/typing.py:746
2024-10-01 19:19:11.686873 Call to __post_init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/guided_fields.py:28 from __init__ in <string>:10
2024-10-01 19:19:11.686899 Return from __post_init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/guided_fields.py:35 to __init__ in <string>:10
2024-10-01 19:19:11.686933 Call to _validate_and_add_requests in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:518 from generate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:323
2024-10-01 19:19:11.686971 Call to _add_guided_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:575 from _validate_and_add_requests in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:548
2024-10-01 19:19:11.687000 Call to get_decoding_config in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/engine/llm_engine.py:758 from _add_guided_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:581
2024-10-01 19:19:11.687023 Return from get_decoding_config in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/engine/llm_engine.py:760 to _add_guided_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:581
2024-10-01 19:19:11.687051 Call to get_tokenizer in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:162 from _add_guided_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:586
2024-10-01 19:19:11.687073 Return from get_tokenizer in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:164 to _add_guided_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:586
2024-10-01 19:19:11.687095 Call to get_local_guided_decoding_logits_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/__init__.py:34 from _add_guided_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/entrypoints/llm.py:584
2024-10-01 19:19:11.687123 Call to get_local_outlines_guided_decoding_logits_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:82 from get_local_guided_decoding_logits_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/__init__.py:40
2024-10-01 19:19:11.687152 Call to _get_guide_and_mode in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:100 from get_local_outlines_guided_decoding_logits_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:92
2024-10-01 19:19:11.687182 Call to <listcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:119 from _get_guide_and_mode in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:119
2024-10-01 19:19:11.687215 Return from <listcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:119 to _get_guide_and_mode in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:119
2024-10-01 19:19:11.687243 Return from _get_guide_and_mode in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:123 to get_local_outlines_guided_decoding_logits_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:92
2024-10-01 19:19:11.687269 Call to _get_logits_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:134 from get_local_outlines_guided_decoding_logits_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:96
2024-10-01 19:19:11.687307 Call to __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_logits_processors.py:96 from _get_logits_processor in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py:141
2024-10-01 19:19:11.687344 Call to wrapper in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:114 from __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/vllm/model_executor/guided_decoding/outlines_logits_processors.py:108
2024-10-01 19:19:11.687374 Call to __cache_key__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:127 from wrapper in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:118
2024-10-01 19:19:11.687403 Call to args_to_key in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:387 from __cache_key__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:129
2024-10-01 19:19:11.687429 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398 from args_to_key in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398
2024-10-01 19:19:11.687450 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398 to args_to_key in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398
2024-10-01 19:19:11.687468 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398 from args_to_key in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398
2024-10-01 19:19:11.687486 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398 to args_to_key in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398
2024-10-01 19:19:11.687504 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398 from args_to_key in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398
2024-10-01 19:19:11.687522 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398 to args_to_key in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398
2024-10-01 19:19:11.687539 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398 from args_to_key in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398
2024-10-01 19:19:11.687561 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398 to args_to_key in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:398
2024-10-01 19:19:11.687587 Return from args_to_key in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:414 to __cache_key__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:129
2024-10-01 19:19:11.687605 Return from __cache_key__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:129 to wrapper in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:118
2024-10-01 19:19:11.687642 Call to get in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:1123 from wrapper in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:119
2024-10-01 19:19:11.687678 Call to put in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:19 from get in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:1149
2024-10-01 19:19:11.687709 Call to dumps in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1464 from put in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:20
2024-10-01 19:19:11.687745 Call to __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1253 from dumps in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1478
2024-10-01 19:19:11.687784 Return from __init__ in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1261 to dumps in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1478
2024-10-01 19:19:11.687807 Call to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1243 from dumps in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1479
2024-10-01 19:19:11.687836 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.687863 Call to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1028 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.687894 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.687921 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.687947 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.687971 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.688014 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.688036 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.688070 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.688088 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.688106 Return from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1040 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.688124 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.688154 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.688180 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.688250 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.688273 Call to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1028 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.688295 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.688315 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.688339 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.688360 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.688407 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:298 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.688433 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:245 to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.688454 Call to _dynamic_class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1001 from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1039
2024-10-01 19:19:11.688478 Call to _class_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:647 from _dynamic_class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1020
2024-10-01 19:19:11.688504 Call to _get_bases in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:608 from _class_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:659
2024-10-01 19:19:11.688527 Return from _get_bases in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:619 to _class_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:659
2024-10-01 19:19:11.688548 Call to _get_or_create_tracker_id in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:108 from _class_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:661
2024-10-01 19:19:11.688674 Return from _get_or_create_tracker_id in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:115 to _class_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:661
2024-10-01 19:19:11.688696 Return from _class_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:656 to _dynamic_class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1020
2024-10-01 19:19:11.688721 Call to _class_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:728 from _dynamic_class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1021
2024-10-01 19:19:11.688750 Call to _extract_class_dict in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:410 from _class_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:729
2024-10-01 19:19:11.688828 Return from _extract_class_dict in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:429 to _class_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:729
2024-10-01 19:19:11.688855 Return from _class_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:764 to _dynamic_class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1021
2024-10-01 19:19:11.688876 Return from _dynamic_class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1018 to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1039
2024-10-01 19:19:11.688897 Return from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1039 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.688920 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.688941 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.688967 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.688987 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.689035 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.689062 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.689084 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.689115 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.689136 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.689162 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.689180 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.689199 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1206 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.689217 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.689241 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.689263 Call to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1028 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.689289 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.689313 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.689336 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.689357 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.689386 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.689407 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.689429 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.689448 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.689467 Return from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1040 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.689486 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.689509 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.689531 Call to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1028 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.689551 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.689572 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.689600 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.689621 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.689657 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.689678 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.689703 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.689721 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.689740 Return from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1040 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.689757 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.689795 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.689818 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.689837 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.689857 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.689880 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.689900 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.689938 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.689959 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.689984 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.690003 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.690022 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1206 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.690040 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.690063 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.690086 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.690108 Call to _property_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:993 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.690132 Return from _property_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:994 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.690151 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.690172 Call to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1028 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.690193 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.690214 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.690236 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.690261 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.690289 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.690310 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.690332 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.690350 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.690369 Return from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1040 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.690388 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.690409 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.690432 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.690451 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.690471 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.690495 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.690516 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.690552 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:298 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.690576 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:245 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.690600 Call to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1190 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208
2024-10-01 19:19:11.690624 Call to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1210 from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1192
2024-10-01 19:19:11.690675 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.690698 Call to _make_empty_cell in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:507 from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.690720 Return from _make_empty_cell in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:513 to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.690741 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.690760 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.690779 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.690798 Return from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1241 to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1192
2024-10-01 19:19:11.690819 Call to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:689 from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1193
2024-10-01 19:19:11.690856 Call to _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:304 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:707
2024-10-01 19:19:11.690883 Call to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311 from _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.690904 Call to _walk_global_ops in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:402 from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.691030 Return from _walk_global_ops in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:404 to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.691056 Return from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311 to _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.691085 Return from _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:326 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:707
2024-10-01 19:19:11.691107 Call to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708
2024-10-01 19:19:11.691127 Return from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708
2024-10-01 19:19:11.691152 Call to _get_cell_contents in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:468 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:711
2024-10-01 19:19:11.691175 Return from _get_cell_contents in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:470 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:711
2024-10-01 19:19:11.691202 Call to _find_imported_submodules in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:329 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:719
2024-10-01 19:19:11.691233 Return from _find_imported_submodules in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:375 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:719
2024-10-01 19:19:11.691256 Return from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:725 to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1193
2024-10-01 19:19:11.691276 Return from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1194 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208
2024-10-01 19:19:11.691294 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.691312 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.691332 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.691356 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.691379 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.691400 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.691423 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.691443 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.691473 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.691494 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.691519 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.691538 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.691558 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1206 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.691576 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.691599 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.691621 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.691645 Call to _code_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:799 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.691688 Return from _code_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:894 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.691707 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.691733 Call to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1028 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.691754 Return from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1037 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.691772 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.691791 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.691813 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.691832 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.691852 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.691874 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.691895 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.691923 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.691944 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.691969 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.691988 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.692006 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1206 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.692028 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692054 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692080 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692101 Call to _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:897 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692124 Return from _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:902 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692142 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692164 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.692183 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.692203 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.692226 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.692246 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.692274 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.692294 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.692318 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.692336 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.692359 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1206 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.692377 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692399 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692421 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.692440 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.692460 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.692482 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.692502 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.692530 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.692551 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.692576 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.692594 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.692613 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1206 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.692631 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692659 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692683 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692703 Call to _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:897 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692723 Return from _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:904 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692742 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.692764 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.692782 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.692802 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.692843 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.692865 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.692896 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.692919 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.692946 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.692966 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.692986 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1206 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.693035 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.693065 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.693090 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.693112 Call to _property_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:993 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.693133 Return from _property_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:994 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.693154 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.693179 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.693200 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.693222 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.693249 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.693272 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.693311 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:298 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.693333 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:245 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.693354 Call to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1190 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208
2024-10-01 19:19:11.693376 Call to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1210 from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1192
2024-10-01 19:19:11.693412 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.693434 Call to _make_empty_cell in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:507 from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.693455 Return from _make_empty_cell in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:513 to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.693475 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.693495 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.693515 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.693538 Return from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1241 to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1192
2024-10-01 19:19:11.693560 Call to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:689 from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1193
2024-10-01 19:19:11.693595 Call to _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:304 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:707
2024-10-01 19:19:11.693622 Call to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311 from _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.693644 Call to _walk_global_ops in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:402 from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.693737 Return from _walk_global_ops in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:404 to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.693761 Return from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311 to _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.693790 Return from _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:326 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:707
2024-10-01 19:19:11.693817 Call to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708
2024-10-01 19:19:11.693840 Return from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708
2024-10-01 19:19:11.693867 Call to _get_cell_contents in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:468 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:711
2024-10-01 19:19:11.693888 Return from _get_cell_contents in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:470 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:711
2024-10-01 19:19:11.693914 Call to _find_imported_submodules in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:329 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:719
2024-10-01 19:19:11.693944 Return from _find_imported_submodules in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:375 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:719
2024-10-01 19:19:11.693968 Return from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:725 to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1193
2024-10-01 19:19:11.693990 Return from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1194 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208
2024-10-01 19:19:11.694020 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.694037 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694056 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694079 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694099 Call to _code_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:799 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694133 Return from _code_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:894 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694164 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694191 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694210 Call to _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:897 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694231 Return from _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:902 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694252 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694274 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694293 Call to _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:897 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694312 Return from _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:904 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694333 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694355 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694374 Call to _property_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:993 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694393 Return from _property_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:994 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694411 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.694433 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.694451 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.694473 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.694501 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.694521 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.694557 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:298 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.694577 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:245 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.694595 Call to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1190 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208
2024-10-01 19:19:11.694615 Call to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1210 from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1192
2024-10-01 19:19:11.694642 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.694662 Call to _make_empty_cell in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:507 from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.694681 Return from _make_empty_cell in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:513 to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.694699 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.694717 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.694735 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.694755 Return from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1241 to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1192
2024-10-01 19:19:11.694775 Call to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:689 from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1193
2024-10-01 19:19:11.694805 Call to _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:304 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:707
2024-10-01 19:19:11.694833 Call to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311 from _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.694852 Call to _walk_global_ops in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:402 from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.694923 Return from _walk_global_ops in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:404 to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.694944 Return from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311 to _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.694969 Return from _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:326 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:707
2024-10-01 19:19:11.694989 Call to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708
2024-10-01 19:19:11.695008 Return from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708
2024-10-01 19:19:11.695031 Call to _get_cell_contents in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:468 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:711
2024-10-01 19:19:11.695050 Return from _get_cell_contents in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:470 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:711
2024-10-01 19:19:11.695073 Call to _find_imported_submodules in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:329 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:719
2024-10-01 19:19:11.695098 Return from _find_imported_submodules in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:375 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:719
2024-10-01 19:19:11.695120 Return from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:725 to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1193
2024-10-01 19:19:11.695140 Return from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1194 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208
2024-10-01 19:19:11.695157 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.695180 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695200 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695222 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695242 Call to _code_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:799 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695273 Return from _code_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:894 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695294 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695315 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695335 Call to _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:897 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695356 Return from _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:902 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695376 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695398 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695417 Call to _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:897 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695436 Return from _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:904 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695456 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695483 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695531 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695561 Call to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1028 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.695583 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.695604 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.695628 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.695649 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.695682 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.695704 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.695727 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.695745 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.695764 Return from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1040 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.695782 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695808 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695831 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695861 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.695890 Call to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1196 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.695909 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.695929 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.695952 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.695972 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.696006 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:298 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.696026 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:245 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1205
2024-10-01 19:19:11.696044 Call to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1190 from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208
2024-10-01 19:19:11.696067 Call to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1210 from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1192
2024-10-01 19:19:11.696094 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.696113 Call to _make_empty_cell in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:507 from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.696133 Return from _make_empty_cell in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:513 to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.696151 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.696169 Call to <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.696188 Return from <genexpr> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239 to _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1239
2024-10-01 19:19:11.696215 Return from _function_getnewargs in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1241 to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1192
2024-10-01 19:19:11.696235 Call to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:689 from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1193
2024-10-01 19:19:11.696265 Call to _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:304 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:707
2024-10-01 19:19:11.696297 Call to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311 from _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.696316 Call to _walk_global_ops in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:402 from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.696389 Return from _walk_global_ops in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:404 to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.696410 Return from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311 to _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:311
2024-10-01 19:19:11.696436 Return from _extract_code_globals in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:326 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:707
2024-10-01 19:19:11.696457 Call to <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708
2024-10-01 19:19:11.696477 Return from <dictcomp> in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:708
2024-10-01 19:19:11.696499 Call to _get_cell_contents in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:468 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:711
2024-10-01 19:19:11.696518 Return from _get_cell_contents in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:470 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:711
2024-10-01 19:19:11.696542 Call to _find_imported_submodules in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:329 from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:719
2024-10-01 19:19:11.696568 Return from _find_imported_submodules in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:375 to _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:719
2024-10-01 19:19:11.696595 Return from _function_getstate in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:725 to _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1193
2024-10-01 19:19:11.696615 Return from _dynamic_function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1194 to _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208
2024-10-01 19:19:11.696633 Return from _function_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1208 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330
2024-10-01 19:19:11.696652 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1330 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696671 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696693 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696714 Call to _code_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:799 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696746 Return from _code_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:894 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696767 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696789 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696810 Call to _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:897 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696835 Return from _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:902 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696856 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696879 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696900 Call to _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:897 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696919 Return from _cell_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:904 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696949 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.696975 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.925063 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.925165 Call to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1028 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.925198 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.925225 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.925261 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.925285 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.925326 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.925349 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.925376 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.925395 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.925414 Return from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1040 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.925432 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.925475 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.925500 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.925563 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.925585 Call to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1028 from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.925607 Call to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:228 from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.925628 Call to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:264 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.925652 Call to _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:195 from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.925673 Return from _whichmodule in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:207 to _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:273
2024-10-01 19:19:11.925705 Return from _lookup_module_and_qualname in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:301 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:243
2024-10-01 19:19:11.925726 Call to _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:181 from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.925751 Return from _is_registered_pickle_by_value in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:192 to _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247
2024-10-01 19:19:11.925769 Return from _should_pickle_by_reference in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:247 to _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1038
2024-10-01 19:19:11.925787 Return from _class_reduce in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1040 to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328
2024-10-01 19:19:11.925805 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1328 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.928094 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.928175 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.928233 Call to reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1290 from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.928256 Return from reducer_override in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1334 to dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245
2024-10-01 19:19:11.928314 Return from dump in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1245 to dumps in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1479
2024-10-01 19:19:11.928342 Return from dumps in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/cloudpickle/cloudpickle.py:1480 to put in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:20
2024-10-01 19:19:11.928392 Call to put in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:139 from put in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:21
2024-10-01 19:19:11.928425 Return from put in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:150 to put in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:21
2024-10-01 19:19:11.928445 Return from put in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/outlines/caching.py:21 to get in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:1149
2024-10-01 19:19:11.928499 Call to _sql in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:646 from get in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:1165
2024-10-01 19:19:11.928528 Call to _con in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:608 from _sql in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:648
2024-10-01 19:19:11.928566 Return from _con in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:644 to _sql in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:648
2024-10-01 19:19:11.928589 Return from _sql in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:648 to get in /home/xz479/miniconda3/envs/conformity/lib/python3.9/site-packages/diskcache/core.py:1165
youkaichao commented 6 hours ago

from the log, it seems the last time you see vllm code, is vllm/model_executor/guided_decoding/outlines_decoding.py .

did you try to remove the guided decoding?

outputs = llm.generate(data, sampling_params, guided_options_request=dict(guided_choice=["agree", "disagree"]))

to

outputs = llm.generate(data, sampling_params)

it is just to identify the root cause.

SpaceHunterInf commented 5 hours ago

Ah! You are absolutely correct, by removing the guided_choice, my local version can generate utterances normally. But, why? What could cause my local machine fail to do the guided_choice constrained generation?

youkaichao commented 4 hours ago

can be an outlier issue. see https://github.com/vllm-project/vllm/pull/7831 for example.

and https://github.com/vllm-project/vllm/pull/7831#issuecomment-2308129931 for changing the guided decoding backend.