Add long recordings with linear drift (20 um/80min): {8,16} channels x {5,10,20,40,80} min

[jamesjun]

Modify from paired_english script

[jamesjun]

1. Generate 80-minute long recordings using eMouse simulator on Kilosort2. Create a linear drift of 20 um during the recording duration. 64-channel layout.
2. Cut recordings to create 5, 10, 20, 40, 80-minute duration recordings and 8 channels (8 recordings) and 16 channels (7 recordings with overlaps).
3. Update spikeforest_recordings
4. Update spikeforest staging
5. Push to the production website
6. Plot the accuracy vs length for various sorters.

[jamesjun]

Step 1 is done, irc2 emouse2mda dir_in dir_out command is added to ironclust

[jamesjun]

Step 2 is done. Error during step 3 while running scripts/upload_long_drift/upload_script.py

Uploading to kachery --- (549.3 MiB): /mnt/ceph/users/jjun/groundtruth/long_drift/1200s_8c/010_synth/raw.mda -> http://132.249.245.245:24342/set/sha1/91d96bfda0afb7923e059a26a467766544727ede?channel
=neuro&signature=55cf8d822d84628eae78a58128826dd754ce2a33
Traceback (most recent call last):
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/util/retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1254, in request
    conn.request(method, url, **httplib_request_kw)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 788, in _http_post_file_data
    req = requests.post(url, data=f)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "upload_script.py", line 107, in <module>
    study_name = study_name
  File "upload_script.py", line 87, in register_study
    register_recording(recdir=path_rec1, output_fname=os.path.join(path_to, rec1+'.json'), label=rec1)
  File "upload_script.py", line 26, in register_recording
    raw_path = ka.store_file(recdir + '/raw.mda')
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 469, in store_file
    _upload_local_file(path, algorithm=algorithm, hash=hash0, config=config)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 657, in _upload_local_file
    resp_obj = _http_post_file_data(url0, path)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 790, in _http_post_file_data
    raise Exception('Error posting file data.')

[jamesjun]

Same error appeared again when I reran scripts/upload_long_drift/upload_script.py

Uploading 006_synth
Computing sha1 of /mnt/ceph/users/jjun/groundtruth/long_drift/1200s_8c/006_synth/raw.mda
Uploading to kachery --- (549.3 MiB): /mnt/ceph/users/jjun/groundtruth/long_drift/1200s_8c/006_synth/raw.mda -> http://132.249.245.245:24342/set/sha1/7982913c20c385a679b589a81ed6cb8aeab74977?channel
=neuro&signature=97fa65dea01ccd5866074ee003d257bc9da6cabf
Traceback (most recent call last):
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:                                                                                                                           [23/340]

Traceback (most recent call last):
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/util/retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 788, in _http_post_file_data
    req = requests.post(url, data=f)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "upload_script.py", line 107, in <module>
    study_name = study_name
  File "upload_script.py", line 77, in register_study
    directory=ka.store_dir(path_rec1),
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 499, in store_dir
    X = read_dir(dirpath, recursive=True, git_annex_mode=git_annex_mode, store_file=True, config=config)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 537, in read_dir
    return _read_file_system_dir(path, recursive=recursive, include_hashes=True, store_files=store_file, git_annex_mode=git_annex_mode, config=config)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 839, in _read_file_system_dir
    store_file(path0, git_annex_mode=git_annex_mode, config=config)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 469, in store_file
    _upload_local_file(path, algorithm=algorithm, hash=hash0, config=config)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 657, in _upload_local_file
    resp_obj = _http_post_file_data(url0, path)
  File "/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py", line 790, in _http_post_file_data
    raise Exception('Error posting file data.')
Exception: Error posting file data.

The IP and port appears to be open.

(spikeforest2) jjun@ccmlin000:upload_long_drift$ nc -vz 132.249.245.245 24342
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 132.249.245.245:24342.
Ncat: 0 bytes sent, 0 bytes received in 0.08 seconds.

[magland]

Strange. Does this error occur immediately, or after a period of time?

Could you try this from the command-line?

kachery-store --to default_readwrite /mnt/ceph/users/jjun/groundtruth/long_drift/1200s_8c/006_synth/raw.mda

If that doesn't work, try with a smaller file.

[jamesjun]

I get the same error below:

(spikeforest2) jjun@ccmlin000:spikeforest_recordings$ kachery-store --to
default_readwrite
/mnt/ceph/users/jjun/groundtruth/long_drift/1200s_8c/006_synth/raw.mda

Uploading to kachery --- (549.3 MiB):
/mnt/ceph/users/jjun/groundtruth/long_drift/1200s_8c/006_synth/raw.mda ->
http://132.249.245.245:24342/set/sha1/7982913c20c385a679b589a81ed6cb8aeab74977?channel=neuro&signature=97fa65dea01ccd5866074ee003d257bc9da6cabf
Traceback (most recent call last):
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py",
line 600, in urlopen
    chunked=chunked)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py",
line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line
1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line
1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line
1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line
1075, in _send_output
    self.send(chunk)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line
996, in send
    self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/adapters.py",
line 449, in send
    timeout=timeout
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py",
line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/util/retry.py",
line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/packages/six.py",
line 685, in reraise
    raise value.with_traceback(tb)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py",
line 600, in urlopen
    chunked=chunked)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/urllib3/connectionpool.py",
line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line
1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line
1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line
1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line
1075, in _send_output
    self.send(chunk)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/http/client.py", line
996, in send
    self.sock.sendall(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.',
BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py",
line 788, in _http_post_file_data
    req = requests.post(url, data=f)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/api.py",
line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/api.py",
line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/sessions.py",
line 533, in request
    resp = self.send(prep, **send_kwargs)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/sessions.py",
line 646, in send
    r = adapter.send(request, **kwargs)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/requests/adapters.py",
line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.',
BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/home/jjun/conda/envs/spikeforest2/bin/kachery-store", line 44,
in <module>
    main()
  File "/mnt/home/jjun/conda/envs/spikeforest2/bin/kachery-store", line 30,
in main
    address = ka.store_file(path)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py",
line 469, in store_file
    _upload_local_file(path, algorithm=algorithm, hash=hash0, config=config)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py",
line 657, in _upload_local_file
    resp_obj = _http_post_file_data(url0, path)
  File
"/mnt/home/jjun/conda/envs/spikeforest2/lib/python3.6/site-packages/kachery/core.py",
line 790, in _http_post_file_data
    raise Exception('Error posting file data.')
Exception: Error posting file data.

-James

On Tue, Mar 31, 2020 at 12:02 PM Jeremy Magland notifications@github.com wrote:

Strange. Does this error occur immediately, or after a period of time?

Could you try this from the command-line?

kachery-store --to default_readwrite /mnt/ceph/users/jjun/groundtruth/long_drift/1200s_8c/006_synth/raw.mda

If that doesn't work, try with a smaller file.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/flatironinstitute/spikeforest_recordings/issues/2#issuecomment-606720501, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACEHBOBR7QGRQVWGIISKSETRKIHYVANCNFSM4LF5RK4Q .

[jamesjun]

Step 3 is done. Step 4 stuck. Upload to staging script worked but "LONG_DRIFT" does not show up.