search

cedadev / cmip6-object-store

CMIP6 Object Store Library
BSD 3-Clause "New" or "Revised" License
4 stars 4 forks source link

Can we fix "Connection pool is full" warning? #14

Closed agstephens closed 4 years ago

agstephens commented 4 years ago

Warning is:

WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ceda-archive-o.s3.jc.rl.ac.uk
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ceda-archive-o.s3.jc.rl.ac.uk
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ceda-archive-o.s3.jc.rl.ac.uk
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ceda-archive-o.s3.jc.rl.ac.uk

I think they retry, so maybe no actual error.

This looks like it might solve things:

        self._fs = s3fs.S3FileSystem(anon=False, secret=self._creds['secret'], 
                                key=self._creds['token'],
                                client_kwargs={'endpoint_url': creds['endpoint_url']},
                                config_kwargs={'max_pool_connections': 5})
agstephens commented 4 years ago

Retry seems to have resolved this.