2p-preprocess update dataset crash

[yiran25]

2p-preprocess pipeline crashes when trying to update suite2p path on flexilims, with error TypeError: Object of type datetime is not JSON serializable. I suspect it's due to an entry called data_proc in ops which is a type of datetime.

[yiran25]

Now the datetime error is solved if I convert datetime to string, but there seems to be a very long string now which caused another new error. I checked all values that are string and they are never that long? I wonder if that refers to the total length of ops converted to JSON or something?

Traceback (most recent call last):
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/bin/preprocess2p", line 33, in <module>
    sys.exit(load_entry_point('2p-preprocess', 'console_scripts', 'preprocess2p')())
  File "/nemo/lab/znamenskiyp/home/users/hey2/codes/duplicates/2p-preprocess/src/preprocess2p/preprocess2p.py", line 551, in entry_point
    defopt.run(main)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/site-packages/defopt.py", line 356, in run
    return call()
  File "/nemo/lab/znamenskiyp/home/users/hey2/codes/duplicates/2p-preprocess/src/preprocess2p/preprocess2p.py", line 523, in main
    suite2p_dataset = run_extraction(flz_session, project, session_name, conflicts, ops)
  File "/nemo/lab/znamenskiyp/home/users/hey2/codes/duplicates/2p-preprocess/src/preprocess2p/preprocess2p.py", line 268, in run_extraction
    suite2p_dataset.update_flexilims(mode="overwrite")
  File "/nemo/lab/znamenskiyp/home/users/hey2/codes/flexiznam/flexiznam/schema/datasets.py", line 436, in update_flexilims
    resp = flz.add_dataset(
  File "/nemo/lab/znamenskiyp/home/users/hey2/codes/flexiznam/flexiznam/main.py", line 626, in add_dataset
    resp = flexilims_session.post(
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/site-packages/flexilims/main.py", line 280, in post
    rep = self.session.post(self.base_url + address, json=json_data)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/site-packages/requests/sessions.py", line 635, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/http/client.py", line 1050, in _send_output
    self.send(chunk)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/http/client.py", line 972, in send
    self.sock.sendall(data)
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/camp/home/hey2/.conda/envs/2p-preprocess_test2/lib/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes

[ablot]

These are all the arrays uploaded by this run_suite2p:

first_tiffs shape: (143,)
frames_per_folder shape: (6,)
frames_per_file shape: (143,)
meanImg shape: (1024, 1024)
refImg shape: (1024, 1024)
yoff shape: (140729,)
xoff shape: (140729,)
corrXY shape: (140729,)
yoff1 shape: (140729, 144)
xoff1 shape: (140729, 144)
corrXY1 shape: (140729, 144)
badframes shape: (140729,)
spatscale_pix shape: (1,)
meanImgE shape: (1024, 1024)
tPC shape: (2000, 30)
regPC shape: (2, 30, 936, 1004)
regDX shape: (30, 3)
max_proj shape: (936, 1004)
Vmax shape: (5000,)
ihop shape: (5000,)
Vsplit shape: (5000,)
Vcorr shape: (936, 1004)
Vmap shape: (5,)

A bunch of them are a lot too big IMO we should not upload them.

[ablot]

Problem was mostly upstream. Closing for now