search

big-data-europe / docker-hadoop

Apache Hadoop docker image
2.18k stars 1.27k forks source link

Can't push data to datanode with webhdfs #144

Open GuillaumeDrillaud opened 1 year ago

GuillaumeDrillaud commented 1 year ago

Here is what I can do :

Here are what I can't do :

To summarize, I can access namenode with anything, but i can't access datanode with webhdfs.

Here are some errors i get : 

gdrillaud@nanw91137:~$ curl -v -i -X PUT -T /export/home/workingArea/data/platform/storage/storage_api/fastAPI/data/to_send/LN-1803_PN-5328_PI-1_SN-1936876_Year-2022_Jday-234_00-00UTC_01.segd  "http://10.20.141.17:9870/webhdfs/v1/Basalt-Files/?op=CREATE"
*   Trying 10.20.141.17:9870...
* Connected to 10.20.141.17 (10.20.141.17) port 9870 (#0)
> PUT /webhdfs/v1/Basalt-Files/?op=CREATE HTTP/1.1
> Host: 10.20.141.17:9870
> User-Agent: curl/7.74.0
> Accept: */*
> Content-Length: 173520544
> Expect: 100-continue
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
HTTP/1.1 100 Continue

* Mark bundle as not supporting multiuse
< HTTP/1.1 307 Temporary Redirect
HTTP/1.1 307 Temporary Redirect
< Date: Fri, 10 Mar 2023 14:45:10 GMT
Date: Fri, 10 Mar 2023 14:45:10 GMT
< Cache-Control: no-cache
Cache-Control: no-cache
< Expires: Fri, 10 Mar 2023 14:45:10 GMT
Expires: Fri, 10 Mar 2023 14:45:10 GMT
< Date: Fri, 10 Mar 2023 14:45:10 GMT
Date: Fri, 10 Mar 2023 14:45:10 GMT
< Pragma: no-cache
Pragma: no-cache
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-FRAME-OPTIONS: SAMEORIGIN
X-FRAME-OPTIONS: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< Location: http://e26472ec19ff:9864/webhdfs/v1/Basalt-Files/?op=CREATE&namenoderpcaddress=namenode:9900&createflag=&createparent=true&overwrite=false
Location: http://e26472ec19ff:9864/webhdfs/v1/Basalt-Files/?op=CREATE&namenoderpcaddress=namenode:9900&createflag=&createparent=true&overwrite=false
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< Content-Length: 0
Content-Length: 0

* HTTP error before end of send, stop sending
<
* Closing connection 0
hdfscli upload "/export/home/workingArea/data/platform/storage/storage_api/fastAPI/data/to_send/LN-1803_PN-5328_PI-1_SN-1936876_Year-2022_Jday-234_00-00UTC_01.segd" "/Basalt-Files/" --alias=dev
ERROR   Error while uploading. Attempting cleanup.
Traceback (most recent call last):
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 650, in upload
    _upload(path_tuple)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 581, in _upload
    self.write(_temp_path, wrap(reader, chunk_size, progress), **kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 527, in write
    consumer(data)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 516, in consumer
    res = self._request(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 209, in _request
    return self._session.request(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/adapters.py", line 521, in send
    low_conn.endheaders()
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f94877b1910>: Failed to establish a new connection: [Errno 111] Connection refused
ERROR   Unexpected exception.
Traceback (most recent call last):
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/config.py", line 195, in wrapper
    return func(*args, **kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/__main__.py", line 257, in main
    client.upload(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 661, in upload
    raise err
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 650, in upload
    _upload(path_tuple)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 581, in _upload
    self.write(_temp_path, wrap(reader, chunk_size, progress), **kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 527, in write
    consumer(data)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 516, in consumer
    res = self._request(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 209, in _request
    return self._session.request(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/adapters.py", line 521, in send
    low_conn.endheaders()
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f94877b1910>: Failed to establish a new connection: [Errno 111] Connection refused
dedkot01 commented 3 months ago

This answer helped me: https://stackoverflow.com/a/64981792/12023127