ReproNim / containers

Containers "distribution" for reproducible neuroimaging
Apache License 2.0
25 stars 14 forks source link

Add retries for docker hub connections #108

Closed yarikoptic closed 4 months ago

yarikoptic commented 4 months ago

I keep getting emails once in a while with

[INFO    ] bids <- docker trends/gift-bids                                                                                                                                                                                        
Traceback (most recent call last):                                                                                                                                                                                                
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 704, in urlopen                                                                                                                                           
    httplib_response = self._make_request(                                                                                                                                                                                        
                       ^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                        
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request                                                                                                                                     
    self._validate_conn(conn)                                                                                                                                                                                                     
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1045, in _validate_conn                                                                                                                                   
    conn.connect()                                                                                                                                                                                                                
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 414, in connect                                                                                                                                               
    self.sock = ssl_wrap_socket(                                                                                                                                                                                                  
                ^^^^^^^^^^^^^^^^                                                                                                                                                                                                  
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket                                                                                                                                        
    ssl_sock = _ssl_wrap_socket_impl(                                                                                                                                                                                             
               ^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                             
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl                                                                                                                                  
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)                                                                                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                         
  File "/usr/lib/python3.11/ssl.py", line 517, in wrap_socket                                                                                                                                                                     
    return self.sslsocket_class._create(                                                                                                                                                                                          
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                          
  File "/usr/lib/python3.11/ssl.py", line 1075, in _create                                                                                                                                                                        
    self.do_handshake()                                                                                                                                                                                                           
  File "/usr/lib/python3.11/ssl.py", line 1346, in do_handshake                                                                                                                                                                   
    self._sslobj.do_handshake()                                                                                                                                                                                                   
ConnectionResetError: [Errno 104] Connection reset by peer                                                                                                                                                                        

During handling of the above exception, another exception occurred:                                                                                                                                                               

Traceback (most recent call last):                                                                                                                                                                                                
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 489, in send                                                                                                                                                   
    resp = conn.urlopen(                                                                                                                                                                                                          
           ^^^^^^^^^^^^^                                                                                                                                                                                                          
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 788, in urlopen                                                                                                                                           
    retries = retries.increment(                                                                                                                                                                                                  
              ^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                  
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 550, in increment                                                                                                                                             
    raise six.reraise(type(error), error, _stacktrace)                                                                                                                                                                            
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                            
  File "/usr/lib/python3/dist-packages/six.py", line 718, in reraise                                                                                                                                                              
    raise value.with_traceback(tb)                                                                                                                                                                                                
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 704, in urlopen                                                                                                                                           
    httplib_response = self._make_request(                                                                                                                                                                                        
                       ^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                        
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request                                                                                                                                     
    self._validate_conn(conn)                                                                                                                                                                                                     
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1045, in _validate_conn                                                                                                                                   
    conn.connect()                                                                                                                                                                                                                
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 414, in connect                                                                                                                                               
    self.sock = ssl_wrap_socket(                                                                                                                                                                                                  
                ^^^^^^^^^^^^^^^^                                                                                                                                                                                                  
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket                                                                                                                                        
    ssl_sock = _ssl_wrap_socket_impl(                                                                                                                                                                                             
               ^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                             
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl                                                                                                                                  
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)                                                                                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                         
  File "/usr/lib/python3.11/ssl.py", line 517, in wrap_socket                                                                                                                                                                     
    return self.sslsocket_class._create(                                                                                                                                                                                          
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                          
  File "/usr/lib/python3.11/ssl.py", line 1075, in _create                                                                                                                                                                        
    self.do_handshake()                                                                                                                                                                                                           
  File "/usr/lib/python3.11/ssl.py", line 1346, in do_handshake                                                                                                                                                                   
    self._sslobj.do_handshake()                                                                                                                                                                                                   
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))                                                                                                                  

During handling of the above exception, another exception occurred:                                                                                                                                                               

Traceback (most recent call last):                                                                                                                                                                                                
  File "/home/yoh/proj/repronim/containers/scripts/create_singularities", line 458, in <module>                                                                                                                                   
    main()                                                                                                                                                                                                                        
  File "/usr/lib/python3/dist-packages/click/core.py", line 1130, in __call__                                                                                                                                                     
    return self.main(*args, **kwargs)                                                                                                                                                                                             
           ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                             
  File "/usr/lib/python3/dist-packages/click/core.py", line 1055, in main                                                                                                                                                         
    rv = self.invoke(ctx)                                                                                                                                                                                                         
         ^^^^^^^^^^^^^^^^                                                                                                                                                                                                         
  File "/usr/lib/python3/dist-packages/click/core.py", line 1404, in invoke                                                                                                                                                       
    return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                
  File "/usr/lib/python3/dist-packages/click/core.py", line 760, in invoke                                                                                                                                                        
    return __callback(*args, **kwargs)                                                                                                                                                                                            
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                            
  File "/home/yoh/proj/repronim/containers/scripts/create_singularities", line 413, in main                                                                                                                                       
    builder.generate_singularity_for_docker_image(dockerhubid, "bids")                                                                                                                                                            
  File "/home/yoh/proj/repronim/containers/scripts/create_singularities", line 302, in generate_singularity_for_docker_image                                                                                                      
    last_version = self.get_last_docker_version_tag(                                                                                                                                                                              
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                              
  File "/home/yoh/proj/repronim/containers/scripts/create_singularities", line 81, in get_last_docker_version_tag                                                                                                                 
    r = requests.get(f"https://registry.hub.docker.com/v2/repositories/{dh}/tags")                                                                                                                                                
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                
  File "/usr/lib/python3/dist-packages/requests/api.py", line 73, in get                                                                                                                                                          
    return request("get", url, params=params, **kwargs)                                                                                                                                                                           
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                           
  File "/usr/lib/python3/dist-packages/requests/api.py", line 59, in request                                                                                                                                                      
    return session.request(method=method, url=url, **kwargs)                                                                                                                                                                      
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                      
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 587, in request                                                                                                                                                
    resp = self.send(prep, **send_kwargs)                                                                                                                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                         
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 701, in send                                                                                                                                                   
    r = adapter.send(request, **kwargs)                                                                                                                                                                                           
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                           
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 547, in send                                                                                                                                                   
    raise ConnectionError(err, request=request)                                                                                                                                                                                   
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))        

so it seems that we should retry a number of times similarly to how we do in other projects (e.g. up to 10 times exponentially growing spreading into up to 10 minutes; but could be anything reasonable really)