Requests connection error when attempting to add content

[angus-s-mitchell]

I'm unable to secure a connection when adding content. I'm using oauth2, and can print correct username to verify it has authenticated correctly. I've followed existing setup instruction - authough the oauth2 authentication instructions were lacking, as they did not indicate I needed to use my own institution's login URL, but I figured that out eventually.

In the code and output below, I've removed identifying details, replacing with "< tags >"

My short python code is:

from arcgis.gis import GIS
import os

source = GIS("https://<institution>.maps.arcgis.com/", client_id='<client_id>')
print("Successfully logged in as: " + source.properties.user.username)

file_list = os.listdir(r"<data_location>")

print(file_list)

for image in file_list:
    print("uploading " + image)
    image_path = "<data_location>" + image
    source.content.add({'type':'TIF',
                        'title': image,
                        'tags': "<tags>"
                        }, data = image_path, owner = "<owner_name>", folder = "<folder>")

The output I get it:

"Please sign in to your GIS and paste the code that is obtained below.
If a web browser does not automatically open, please navigate to the URL below yourself instead.
Opening web browser to navigate to: <authentication URL>
/usr/lib/python3.10/getpass.py:91: GetPassWarning: Can not control echo on the terminal.
  passwd = fallback_getpass(prompt, stream)
Warning: Password input may be echoed.
Enter code obtained on signing in using SAML: <code>

/home/a/.local/lib/python3.10/site-packages/urllib3/connectionpool.py:981: InsecureRequestWarning: Unverified HTTPS request is being made to host '<institution>.maps.arcgis.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
Successfully logged in as: <username>
<printed list of image files>
uploading <image_file_name>
'https://<institution>.maps.arcgis.comsharing/rest/content/users/<username>/items/<number>/addPart' generated an exception: A connection error has occurred: HTTPSConnectionPool(host='<institution>..maps.arcgis.comsharing', port=443): Max retries exceeded with url: /rest/content/users/<username>/items/<number>/addPart (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efd2de03c70>: Failed to establish a new connection: [Errno -2] Name or service not known'))
'https://<institution>.maps.arcgis.comsharing/rest/content/users/<username>/items/<number>/addPart' generated an exception: A connection error has occurred: HTTPSConnectionPool(host='<institution>..maps.arcgis.comsharing', port=443): Max retries exceeded with url: /rest/content/users/<username>/items/<number>/addPart (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efd2de03c70>: Failed to establish a new connection: [Errno -2] Name or service not known'))
'https://<institution>.maps.arcgis.comsharing/rest/content/users/<username>/items/<number>/addPart' generated an exception: A connection error has occurred: HTTPSConnectionPool(host='<institution>..maps.arcgis.comsharing', port=443): Max retries exceeded with url: /rest/content/users/<username>/items/<number>/addPart (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efd2de03c70>: Failed to establish a new connection: [Errno -2] Name or service not known'))
'https://<institution>.maps.arcgis.comsharing/rest/content/users/<username>/items/<number>/addPart' generated an exception: A connection error has occurred: HTTPSConnectionPool(host='<institution>..maps.arcgis.comsharing', port=443): Max retries exceeded with url: /rest/content/users/<username>/items/<number>/addPart (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efd2de03c70>: Failed to establish a new connection: [Errno -2] Name or service not known'))
'https://<institution>.maps.arcgis.comsharing/rest/content/users/<username>/items/<number>/addPart' generated an exception: A connection error has occurred: HTTPSConnectionPool(host='<institution>..maps.arcgis.comsharing', port=443): Max retries exceeded with url: /rest/content/users/<username>/items/<number>/addPart (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efd2de03c70>: Failed to establish a new connection: [Errno -2] Name or service not known'))
'https://<institution>.maps.arcgis.comsharing/rest/content/users/<username>/items/<number>/addPart' generated an exception: A connection error has occurred: HTTPSConnectionPool(host='<institution>..maps.arcgis.comsharing', port=443): Max retries exceeded with url: /rest/content/users/<username>/items/<number>/addPart (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efd2de03c70>: Failed to establish a new connection: [Errno -2] Name or service not known'))
'https://<institution>.maps.arcgis.comsharing/rest/content/users/<username>/items/<number>/addPart' generated an exception: A connection error has occurred: HTTPSConnectionPool(host='<institution>..maps.arcgis.comsharing', port=443): Max retries exceeded with url: /rest/content/users/<username>/items/<number>/addPart (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efd2de03c70>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Traceback (most recent call last):
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/connection.py", line 309, in connect
    conn = self._new_conn()
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7efd2ded8100>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/a/.local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/home/a/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<institution>.maps.arcgis.comsharing', port=443): Max retries exceeded with url: /rest/content/users/<institution>/items/<number>/commit (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efd2ded8100>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/a/.local/lib/python3.10/site-packages/arcgis/gis/_impl/_con/_connection.py", line 1206, in post
    resp = self._session.post(
  File "/home/a/.local/lib/python3.10/site-packages/arcgis/auth/api.py", line 471, in post
    return self._session.post(
  File "/home/a/.local/lib/python3.10/site-packages/requests/sessions.py", line 635, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/home/a/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/a/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/a/.local/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='<institution>.maps.arcgis.comsharing', port=443): Max retries exceeded with url: /rest/content/users/<username>/items/<number>/commit (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efd2ded8100>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/a/PycharmProjects/native grass classification/batch-upload-to-arcgis.py", line 15, in <module>
    source.content.add({'type':'TIF',
  File "/home/a/.local/lib/python3.10/site-packages/arcgis/gis/__init__.py", line 5281, in add
    status = self._add_by_part(
  File "/home/a/.local/lib/python3.10/site-packages/arcgis/gis/__init__.py", line 4942, in _add_by_part
    res = self._gis._con.post(url, params)
  File "/home/a/.local/lib/python3.10/site-packages/arcgis/gis/_impl/_con/_connection.py", line 1229, in post
    raise requests.exceptions.ConnectionError(
requests.exceptions.ConnectionError: A connection error has occurred: HTTPSConnectionPool(host='<institution>.maps.arcgis.comsharing', port=443): Max retries exceeded with url: /rest/content/users/<username>/items/<number>/commit (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efd2ded8100>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Process finished with exit code 1

I'm working in python 3.10.6. In case it provides any extra clarity, here is the python environment I'm working in.

Package                      Version
---------------------------- -------------
absl-py                      1.3.0
affine                       2.3.1
aiohttp                      3.8.3
aiosignal                    1.3.1
anyio                        3.6.2
appdirs                      1.4.4
apturl                       0.5.2
arcgis                       2.0.0
argon2-cffi                  21.3.0
argon2-cffi-bindings         21.2.0
asttokens                    2.0.8
astunparse                   1.6.3
async-timeout                4.0.2
attrs                        22.1.0
Babel                        2.11.0
backcall                     0.2.0
bcrypt                       3.2.0
beautifulsoup4               4.11.1
beniget                      0.4.1
bleach                       5.0.1
blinker                      1.4
branca                       0.6.0
Brlapi                       0.8.3
Brotli                       1.0.9
cachetools                   5.2.0
certifi                      2020.6.20
cffi                         1.15.1
chardet                      4.0.0
charset-normalizer           2.1.1
click                        8.0.3
click-plugins                1.1.1
cligj                        0.7.2
colorama                     0.4.4
command-not-found            0.3
config                       0.5.1
contourpy                    1.0.5
cryptography                 3.4.8
cupshelpers                  1.0
cycler                       0.11.0
dataframe-image              0.1.5
DateTimeRange                1.2.0
dbus-python                  1.2.18
debugpy                      1.6.3
decorator                    5.1.1
defer                        1.0.6
defusedxml                   0.7.1
distro                       1.7.0
distro-info                  1.1build1
duplicity                    0.8.21
entrypoints                  0.4
executing                    1.1.1
fasteners                    0.14.1
fastjsonschema               2.16.2
Fiona                        1.9a3
flatbuffers                  22.9.24
folium                       0.13.0
fonttools                    4.38.0
frozenlist                   1.3.3
fs                           2.4.12
future                       0.18.2
gast                         0.4.0
GDAL                         3.6.2
geojson                      2.5.0
geomet                       1.0.0
geopandas                    0.12.1
google-auth                  2.13.0
google-auth-oauthlib         0.4.6
google-pasta                 0.2.0
grpcio                       1.50.0
gssapi                       1.8.2
h11                          0.12.0
h5py                         3.7.0
html5lib                     1.1
httpcore                     0.15.0
httplib2                     0.20.2
httpx                        0.23.0
idna                         3.3
imageio                      2.22.4
importlib-metadata           4.6.4
ipykernel                    6.16.2
ipython                      8.5.0
ipython-genutils             0.2.0
ipywidgets                   8.0.2
jedi                         0.18.1
jeepney                      0.7.1
Jinja2                       3.1.2
joblib                       1.2.0
json5                        0.9.11
jsonschema                   4.17.3
jupyter                      1.0.0
jupyter_client               7.4.4
jupyter-console              6.4.4
jupyter_core                 4.11.2
jupyter-server               1.21.0
jupyterlab                   3.5.3
jupyterlab-pygments          0.2.2
jupyterlab_server            2.19.0
jupyterlab-widgets           3.0.3
keras                        2.11.0
Keras-Preprocessing          1.1.2
keyring                      21.7.0
kiwisolver                   1.4.4
language-selector            0.1
launchpadlib                 1.10.16
lazr.restfulclient           0.14.4
lazr.uri                     1.0.6
lerc                         0.1.0
libclang                     14.0.6
lockfile                     0.12.2
louis                        3.20.0
lxml                         4.8.0
lz4                          3.1.3+dfsg
macaroonbakery               1.3.1
Mako                         1.1.3
Markdown                     3.4.1
MarkupSafe                   2.1.1
matplotlib                   3.6.1
matplotlib-inline            0.1.6
mbstrdecoder                 1.1.1
mistune                      2.0.4
monotonic                    1.6
more-itertools               8.10.0
mpmath                       0.0.0
multidict                    6.0.4
munch                        2.5.1.dev12
nbclassic                    0.4.5
nbclient                     0.7.0
nbconvert                    7.2.2
nbformat                     5.7.0
nest-asyncio                 1.5.6
netifaces                    0.11.0
networkx                     2.8.8
notebook                     6.5.1
notebook_shim                0.2.0
ntlm-auth                    1.5.0
numpy                        1.23.5
oauthlib                     3.2.0
olefile                      0.46
opencv-python                4.6.0.66
opt-einsum                   3.3.0
OWSLib                       0.27.2
packaging                    21.3
pandas                       1.5.1
pandocfilters                1.5.0
paramiko                     2.9.3
parso                        0.8.3
pbasemap                     0.1.0
pbr                          5.8.0
pexpect                      4.8.0
pickleshare                  0.7.5
Pillow                       9.0.1
pip                          21.3.1
planet                       2.0a5
plotly                       5.4.0
ply                          3.11
progressbar                  2.5
prometheus-client            0.15.0
prompt-toolkit               3.0.31
protobuf                     3.12.4
psutil                       5.9.3
psycopg2                     2.9.2
ptyprocess                   0.7.0
pure-eval                    0.2.2
pyasn1                       0.4.8
pyasn1-modules               0.2.8
pycairo                      1.20.1
pycparser                    2.21
pycups                       2.0.1
Pygments                     2.13.0
PyGObject                    3.42.1
PyJWT                        2.3.0
pymacaroons                  0.13.0
PyNaCl                       1.5.0
pyparsing                    2.4.7
pyproj                       3.4.0
PyQt5                        5.15.6
PyQt5-sip                    12.9.1
pyRFC3339                    1.1
pyrsistent                   0.18.1
pyshp                        2.3.1
pySmartDL                    1.3.4
python-apt                   2.4.0
python-certifi-win32         1.6.1
python-dateutil              2.8.2
python-debian                0.1.43ubuntu1
pythran                      0.10.0
pytz                         2022.6
PyWavelets                   1.4.1
pyxdg                        0.27
PyYAML                       5.4.1
pyzmq                        24.0.1
QScintilla                   2.11.6
qtconsole                    5.3.2
QtPy                         2.2.1
rasterio                     1.3.4
reportlab                    3.6.8
requests                     2.28.2
requests-futures             0.9.9
requests-gssapi              1.2.3
requests-ntlm                1.1.0
requests-oauthlib            1.3.1
requests-toolbelt            0.10.1
retrying                     1.3.3
rfc3986                      1.5.0
rsa                          4.9
scikit-image                 0.18.3
scikit-learn                 1.1.3
scipy                        1.8.0
SecretStorage                3.3.1
Send2Trash                   1.8.0
setuptools                   60.2.0
setuptools-scm               7.1.0
shapely                      2.0b2
six                          1.16.0
sklearn                      0.0.post1
sniffio                      1.3.0
snuggs                       1.4.7
soupsieve                    2.3.2.post1
stack-data                   0.5.1
sympy                        1.9
systemd-python               234
tenacity                     6.3.1
tensorboard                  2.11.0
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.1
tensorflow                   2.11.0
tensorflow-estimator         2.11.0
tensorflow-io-gcs-filesystem 0.27.0
termcolor                    2.0.1
terminado                    0.17.0
threadpoolctl                3.1.0
tifffile                     2022.10.10
tinycss2                     1.2.1
tomli                        2.0.1
tornado                      6.2
tqdm                         4.64.1
traitlets                    5.5.0
typepy                       1.3.0
typing_extensions            4.4.0
ubuntu-advantage-tools       27.12
ubuntu-drivers-common        0.0.0
ufoLib2                      0.13.1
ufw                          0.36.1
ujson                        5.7.0
unattended-upgrades          0.1
unicodedata2                 14.0.0
urllib3                      1.25.11
usb-creator                  0.3.7
wadllib                      1.3.6
wcwidth                      0.2.5
webencodings                 0.5.1
websocket-client             1.4.1
Werkzeug                     2.2.2
wheel                        0.37.1
widgetsnbextension           4.0.3
wrapt                        1.14.1
xdg                          5
xkit                         0.0.0
yarl                         1.8.2
zipp                         1.0.0

[hildermesmedeiros]

@angus-s-mitchell , I would try to pass the client_secret. You can check how in my issue https://github.com/Esri/arcgis-python-api/issues/1374

But right now, as you can see here : "https://developers.arcgis.com/python/guide/working-with-different-authentication-schemes/", This won't work in a script workflow.

Long history short, if you pass only client_id, it will prompt an input and redirect you to auth page. If you pass the client_secret, it will work and you will have acess to some routes, but not all, you might need to do others configs in developers page.

[angus-s-mitchell]

@hildermesmedeiros, I'm not seeking to run this code as part of a script workflow. I'm simply trying to use the intended oauth2 functionality as described in the guide you linked - "https://developers.arcgis.com/python/guide/working-with-different-authentication-schemes/". As you can see in the output I posted, I successfully authenticate (according to the guide, printing properties.user.username is how we confirm authentication.) If this is the case, the issue is separate to the initial authorisation - when content.add is called, we get a requests connection error.

[nanaeaubry]

@angus-s-mitchell

The type is not correctly set, it needs to be set to 'Image' since 'TIF' is not a type of service.

gis = GIS(profile="your_online_profile")
gis.content.add(
    {"type": "Image", "title": "test_tif", "tags": ["tif", "test"]},
    data=r"C:\ipython_workfolder\workspace\at3_1m4_01.tif",
)

To see a list of service types refer to: https://developers.arcgis.com/rest/users-groups-and-items/items-and-item-types.htm

Hope this helps :)