search

MouseLand / Kilosort

Fast spike sorting with drift correction
https://kilosort.readthedocs.io/en/latest/
GNU General Public License v3.0
488 stars 247 forks source link

pytest --run slow gives urllib.error.HTTPError: HTTP Error 403: Forbidden #721

Closed Idavr closed 5 months ago

Idavr commented 5 months ago

Describe the issue:

Hello!

I am trying to get KS4 working with the minimal installation as I only am going to use the terminal commands for Kilosorting - with the intention of trying to get it to run on a remote cluster. I have followed all the installation instructions but when I try to run pytest --run slow I end up with a lot of urllib.error.HTTPError: HTTP Error 403: Forbidden messages. The entire terminal for this is pasted in below:

(kilosort) idavalik@ida-HP-Z4-G5-Workstation:~/Kilosort/tests$ pytest --runslow
============================= test session starts ==============================
platform linux -- Python 3.9.19, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/idavalik/Kilosort
configfile: pytest.ini
collected 24 items                                                             

test_clustering.py 
Downloading test data ...
E....                                                 [ 20%]
test_dtype.py EEEEE                                                      [ 41%]
test_full_pipeline.py E                                                  [ 45%]
test_io.py .EEEEE                                                        [ 70%]
test_parameters.py .                                                     [ 75%]
test_preprocessing.py ....E                                              [ 95%]
test_spikedetect.py E                                                    [100%]

==================================== ERRORS ====================================
__________________ ERROR at setup of TestCenters.test_linear ___________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
________________________ ERROR at setup of test_uint16 _________________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
_________________________ ERROR at setup of test_int16 _________________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
_________________________ ERROR at setup of test_int32 _________________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
________________________ ERROR at setup of test_float32 ________________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
______________________ ERROR at setup of test_unsupported ______________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
_______________________ ERROR at setup of test_pipeline ________________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
_____________________ ERROR at setup of test_bat_extension _____________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
_____________________ ERROR at setup of test_dat_extension _____________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
_______________________ ERROR at setup of test_tmin_tmax _______________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
_______________________ ERROR at setup of test_tmin_only _______________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
_______________________ ERROR at setup of test_tmax_only _______________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
______________ ERROR at setup of TestWhitening.test_get_whitening ______________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
______________________ ERROR at setup of test_wpca_wtemp _______________________

download = ''
capture_mgr = <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=5 _state='suspended' tmpfile=<_io....xtIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
gpu = False

    @pytest.fixture(scope='session')
    def data_directory(download, capture_mgr, gpu):
        """Specifies directory for test data and results, downloads if needed."""

        # Set path to directory within tests/ folder dynamically
        data_path = DOWNLOADS_DIR / '.test_data/'
        data_path.mkdir(parents=True, exist_ok=True)

        binary_path = data_path / 'ZFM-02370_mini.imec0.ap.short.bin'
        binary_url = 'https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip'
        if (download == 'binary') or (download == 'both'):
            if binary_path.is_file():
                binary_path.unlink()
        if not binary_path.is_file():
            with capture_mgr.global_and_fixture_disabled():
                print('\nDownloading test data ...')
>               download_data(binary_path, binary_url)

conftest.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:112: in download_data
    download_url_to_file(remote, zip_file)
conftest.py:142: in download_url_to_file
    u = urlopen(url)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:214: in urlopen
    return opener.open(url, data, timeout)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:523: in open
    response = meth(req, response)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:632: in http_response
    response = self.parent.error(
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:561: in error
    return self._call_chain(*args)
../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:494: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x703618db91f0>
req = <urllib.request.Request object at 0x703618db9550>
fp = <http.client.HTTPResponse object at 0x703618db9880>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x703618db9bb0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

../../anaconda3/envs/kilosort/lib/python3.9/urllib/request.py:641: HTTPError
=========================== short test summary info ============================
ERROR test_clustering.py::TestCenters::test_linear - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_dtype.py::test_uint16 - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_dtype.py::test_int16 - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_dtype.py::test_int32 - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_dtype.py::test_float32 - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_dtype.py::test_unsupported - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_full_pipeline.py::test_pipeline - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_io.py::test_bat_extension - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_io.py::test_dat_extension - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_io.py::test_tmin_tmax - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_io.py::test_tmin_only - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_io.py::test_tmax_only - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_preprocessing.py::TestWhitening::test_get_whitening - urllib.error.HTTPError: HTTP Error 403: Forbidden
ERROR test_spikedetect.py::test_wpca_wtemp - urllib.error.HTTPError: HTTP Error 403: Forbidden
======================== 10 passed, 14 errors in 4.34s =========================

OS: Ubuntu 22.04.4 LTS, 64-bit Python: 3.9.19 NVIDIA Corporation: GA 104GL [RTX A4000], driver metapackage from driver-535 Cuda 11.5

I hope you have any insight into this to help me get KS4 working through the terminal.

jacobpennington commented 5 months ago

@Idavr This most likely means the cluster you're using restricts downloads. The tests will automatically download test data and probes if they are not present in the expected directory. You will either need to get around that restriction somehow (you would have to ask whoever manages the cluster about that), or you can manually place the necessary files there before running the tests.

You can get the exact directory for your remote machine by running this code snippet:

from kilosort.utils import DOWNLOADS_DIR
print(DOWNLOADS_DIR)

You can also modify where that directory goes by changing the KILOSORT_LOCAL_DOWNLOADS_PATH environment variable.

Probably the simplest thing to do would be to run the tests on a local machine that doesn't have download restrictions, then copy the downloaded files and folders to that directory on the remote machine. The directory should end up looking like:

Idavr commented 5 months ago

@jacobpennington I tried the same thing on a local computer but with the same results. This is however on a university-network so that might be it... Is there anywhere to manually download these particular test data files so I can move them to the appropriate folders like you suggested?

jacobpennington commented 5 months ago

Here are the URLs. Note you will need to unzip the data and results. I updated my previous comment to show where to put the results, I forgot to include them earlier.

Probes: https://www.kilosort.org/downloads/neuropixPhase3A_kilosortChanMap.mat https://www.kilosort.org/downloads/neuropixPhase3B1_kilosortChanMap.mat https://www.kilosort.org/downloads/neuropixPhase3B2_kilosortChanMap.mat https://www.kilosort.org/downloads/NP2_kilosortChanMap.mat https://www.kilosort.org/downloads/Linear16x1_kilosortChanMap.mat

Data: https://www.kilosort.org/downloads/ZFM-02370_mini.imec0.ap.short.zip

Results: https://www.kilosort.org/downloads/pytest_gpu.zip https://www.kilosort.org/downloads/pytest.zip