Error: HTTP Error 429: Too Many Requests

[lfenske-93]

Hi,

when I try to test your tool I run in the following error:

  warnings.warn(
Downloading Schemes using 40 threads:   0%|                                                                                                                                                                                                       | 0/153 [00:00<?, ?Schemes/s]
Traceback (most recent call last):
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/bin/fastmlst", line 130, in <module>
    update_mlstdb(args.threads)
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/site-packages/fastmlst/update_mlst_kit.py", line 137, in update_mlstdb
    for result in tqdm(t.imap(download_fasta, datadb.items()),
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/multiprocessing/pool.py", line 873, in next
    raise value
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/site-packages/fastmlst/update_mlst_kit.py", line 118, in download_fasta
    urlretrieve(file, str(outdir) + '/' +
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/urllib/request.py", line 240, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
                            ^^^^^^^^^^^^^^^^^^
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/urllib/request.py", line 215, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/urllib/request.py", line 521, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/urllib/request.py", line 630, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/urllib/request.py", line 559, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/vol/gbs_lhl/no_backup/mambaforge/envs/fastmlst/lib/python3.12/urllib/request.py", line 639, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 429: Too Many Requests

I installed it via conda. Do you have any ideas how to fix it?

Greetings, Linda

[EnzoAndree]

I think the web server is blocking the requests. try to use less threads using the -t parameter. e.g. fastmlst -t 2

[szhan]

I encountered the same problem. I have made some progress by setting the number of threads to a lower value (specifically -t 1) as @EnzoAndree suggested above. But I encountered another error. I'm not sure if it is caused by the web server blocking more requests.

Downloading Schemes using 1 threads:  28%|███████████████▍                                       | 43/153 [01:26<03:42,  2.02s/Schemes]
Traceback (most recent call last):
  File "/opt/anaconda3/envs/test/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/opt/anaconda3/envs/test/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/anaconda3/envs/test/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/anaconda3/envs/test/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/anaconda3/envs/test/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/opt/anaconda3/envs/test/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/opt/anaconda3/envs/test/lib/python3.9/http/client.py", line 1447, in connect
    super().connect()
  File "/opt/anaconda3/envs/test/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/opt/anaconda3/envs/test/lib/python3.9/socket.py", line 844, in create_connection
    raise err
  File "/opt/anaconda3/envs/test/lib/python3.9/socket.py", line 832, in create_connection
    sock.connect(sa)
OSError: [Errno 12] Cannot allocate memory

[EnzoAndree]

Hello @szhan,

It looks like you're encountering a memory allocation error, which suggests that your system might not have enough resources available to handle the tasks being executed. This could be due to large data handling that exceed your available memory.

To confirm that the issue is related to memory constraints, you could monitor your system's memory usage while running the script. On Linux, you can use tools like htop or free -m to observe the memory usage in real-time. On Windows, the Task Manager will provide similar insights.

If memory usage is indeed the problem, here are a few suggestions to try:

1. Close other applications and background processes that might be consuming significant memory to free up resources.

2. Increase the physical memory on your machine, if possible, to accommodate more demanding operations.

3. Utilize Swap Space: If you're on Linux, ensuring that you have enough swap space can help as a buffer for when your RAM is fully utilized, though this may slow down process execution due to the slower read/write speed compared to RAM.

Implement these checks and adjustments, and see if the situation improves. Let me know how it goes!

Best regards

[szhan]

Thanks, @EnzoAndree . It is strange, because how far I get in terms of downloading 153 schemes varies from run to run. Once I got as far as 30%, and sometimes I get the above error just after 1 scheme is downloaded. Also, it doesn't seem that the physical memory of my machine is anywhere near drained (I'm using a MacBook Pro with 18 Gb physical memory).

[szhan]

Is this task supposed to consume a lot of memory? If so, it would be nice to mention in the README.

[EnzoAndree]

Hello @szhan,

Thanks for the updates. Typically, downloading schemes as performed by fastmlst should not be highly memory-intensive. This suggests that the issue you're experiencing may be related more to other aspects like settings or environment stability rather than inherent memory demands of the task.

Given the varying progress on each run and the fact that your physical memory does not appear to be near its limit, I recommend trying the following:

1. Reinstall fastmlst: Sometimes, inconsistencies or corruption in the installation can cause unexpected errors. Reinstalling might resolve the issue. You can do this via Conda, making sure to remove the previous environment or installation cleanly before reinstalling.

2. Test in a clean environment: If reinstalling doesn't help, try running the tool in a fresh environment to rule out any conflicts with other packages or system settings.

If these steps don't resolve the problem, it might help us to look deeper into potential bugs or specifics of the implementation that are not directly related to memory issues. Please do keep us informed on how things progress after trying the above steps.

Additionally, if the problem persists after trying these steps, it might be beneficial to open a new issue on GitHub. This will allow us to track the specific problem more effectively and involve others who might be experiencing similar issues or have insights to offer. When opening a new issue, please include all relevant details about the errors, system environment, and any other observations that might help diagnose the problem.

[szhan]

Thanks. I created a fresh environment with Python 3.12 and reinstalled fastmlst, but I am still encountering this problem. I'll create an issue once i get a better handle on what may be happening.

[szhan]

I have managed to download all the schemes when running fastmlst on an older MacBook Pro, which doesn't use Apple Silicon. It seems it may have to do with the architecture.

[szhan]

It turns out that some setup is incorrect with my laptop, and has nothing to do with fastmlst. Thanks for your help @EnzoAndree.