search

HDFGroup / hsds

Cloud-native, service based access to HDF data
https://www.hdfgroup.org/solutions/hdf-kita/
Apache License 2.0
128 stars 52 forks source link

Unable to start HSDS server #127

Closed anotherkhan closed 1 year ago

anotherkhan commented 2 years ago

I am following quick start steps to install HSDS server on my local machine (Windows 10) without docker.

When I execute ./runall.sh --no-docker from Git Bash, I receive following error:

$ ./runall.sh --no-docker
--no_docker option specified - using directory: /tmp/hs for socket and log files
using password file: admin/config/passwd.txt
no AWS or AZURE env set, using POSIX storage
no docker startup
Using posix storage: C:\Users\sami.a.khan\hsds\hsds_data
./runall.sh: line 176: hsds: command not found
jreadey commented 2 years ago

What do you get if you do: $ pip freeze | grep hsds?

anotherkhan commented 2 years ago

Thanks for quick reply.

> pip freeze | findstr hsds
hsds==0.7.0b11
jreadey commented 2 years ago

Thanks for replacing 'grep' with 'findstr'. :)

That's strange. Are you using Anaconda for your python environment? What happens if you just run "hsds" in the Anaconda command window? You should see a hsds message: "Usage: starts hsds.."

anotherkhan commented 2 years ago

Yes I am using Anaconda(miniconda3)

>hsds
usage: Starts hsds a REST-based service for HDF5 data.
hsds: error: one of the arguments --root_dir --bucket_name is required
jreadey commented 2 years ago

If I do a runall.sh from the git bash prompt on Windows 10, it finds hsds, but I get an error: ValueError: Unsupported signal: 2`. So even with the script finding hsds (likely dependent on you've enabled Anaconda for the Git Mash Mingw environment), there are still issues...

The no-docker option uses Python sockets which apparently aren't totally compatible between Windows and Max/Linux. From: https://docs.python.org/3/library/socket.html: "Note Some behavior may be platform dependent, since calls are made to the operating system socket APIs."

So it seems like supporting no-docker on Windows would be a bit of work.

For now, I'd recommend running HSDS on Windows with Docker. I'll try this out now myself with Docker Desktop 4.6.0 and see if any changes are needed to the Docker install guide (https://github.com/HDFGroup/hsds/blob/master/docs/docker_install_posix.md)

anotherkhan commented 2 years ago

Thank you for you help. As recommended I tried running HSDS on Windows with Docker. Docker Desktop version 4.5.1.

After building docker custom image I am unable to start server.

$ ./runall.sh
no AWS or AZURE env set, using POSIX storage
creating directory C:\Users\user\hsds\hsds_data/C:\Users\sami.a.khan\hsds\hsds_data\hsdstest
mkdir: cannot create directory ‘C:\\Users\\user\\hsds\\hsds_data/C:\\Users\\user\\hsds\\hsds_data\\hsdstest’: No such file or directory
Running docker-compose -f admin/docker/docker-compose.posix.yml up
Creating network "hsds_default" with the default driver
Creating hsds_head_1 ...
Creating hsds_head_1 ... done
Creating hsds_sn_1   ...
Creating hsds_rangeget_1 ...
Creating hsds_sn_1       ... done
Creating hsds_rangeget_1 ... done
Creating hsds_dn_1       ...
Creating hsds_dn_2       ...
Creating hsds_dn_3       ...
Creating hsds_dn_4       ...
Creating hsds_dn_1       ... done
Creating hsds_dn_3       ... done
Creating hsds_dn_2       ... done
Creating hsds_dn_4       ... done
1: waiting for server startup (status: 000)
2: waiting for server startup (status: 000)
3: waiting for server startup (status: 000)
4: waiting for server startup (status: 000)
jreadey commented 2 years ago

Thanks for trying with Docker Desktop. I wasn't successful myself -- my Windows box is a bit old and the cpu doesn't have the hyper-v support for Docker Desktop.

With the runall.sh, it looks like the script is not doing the right thing with the windows file paths. In the Git Bash shell, try setting the ROOT_DIR using the equivalent Linux path. E.g.: $ export ROOT_DIR=/c/Users/sami.a.khan/hsds/hsds_data/hsdstest and then do a runall.sh.

I guess it would be best if there was a windows bat script that you could run from the windows cmd prompt to start the server. Would be more than happy if you'd like to contribute it!

BTW, I'm curious if anyone has tried running HSDS on windows using WSL2 (windows subsystem for Linux). By all accounts (see https://medium.com/swlh/data-science-using-wsl2-seamless-integration-of-ubuntu-and-windows-10-dbe94cbac2b6) its very convenient for running Linux apps on Windows. I expect the --no-docker option would "just work" on WSL2, but don't have a way to test it myself.

anotherkhan commented 2 years ago

Thanks, I set ROOT_DIR path mentioned above and then executed runall.sh in GIT bash. No issues with as earlier with creating a directory but stuck on waiting for server to startup.

$ ./runall.sh
no AWS or AZURE env set, using POSIX storage
creating directory /c/Users/sami.a.khan/hsds/hsds_data/hsdstest/hsdstest
Running docker-compose -f admin/docker/docker-compose.posix.yml up
Recreating hsds_head_1 ...
Recreating hsds_head_1 ... done
Recreating hsds_rangeget_1 ...
Recreating hsds_sn_1       ...
Recreating hsds_rangeget_1 ... done
Recreating hsds_dn_1       ...
Recreating hsds_dn_2       ...
Recreating hsds_dn_3       ...
Recreating hsds_dn_4       ...
Recreating hsds_sn_1       ... done
Recreating hsds_dn_3       ... done
Recreating hsds_dn_1       ... done
Recreating hsds_dn_2       ... done
Recreating hsds_dn_4       ... done
1: waiting for server startup (status: 000)
2: waiting for server startup (status: 000)
3: waiting for server startup (status: 000)
4: waiting for server startup (status: 000)
5: waiting for server startup (status: 000)
6: waiting for server startup (status: 000)
7: waiting for server startup (status: 000)
jreadey commented 2 years ago

Are the containers starting? Run $docker ps to get status.

If so, run $docker logs hsds_sn_1 and check to see if any errors are showing up.

anotherkhan commented 2 years ago

Please see the logs from above docker command execution

>docker ps
CONTAINER ID   IMAGE           COMMAND                  CREATED              STATUS                            PORTS     NAMES
43820fde4d8a   hdfgroup/hsds   "/bin/bash -c 'sourc…"   About a minute ago   Restarting (126) 2 seconds ago              hsds_dn_2
9db32e14909e   hdfgroup/hsds   "/bin/bash -c 'sourc…"   About a minute ago   Restarting (126) 13 seconds ago             hsds_dn_4
ba9a666a9226   hdfgroup/hsds   "/bin/bash -c 'sourc…"   About a minute ago   Restarting (126) 10 seconds ago             hsds_dn_1
7157e91555dc   hdfgroup/hsds   "/bin/bash -c 'sourc…"   About a minute ago   Restarting (126) 10 seconds ago             hsds_dn_3
4ede901fc00d   hdfgroup/hsds   "/bin/bash -c 'sourc…"   About a minute ago   Restarting (126) 8 seconds ago              hsds_rangeget_1
426cfb88f763   hdfgroup/hsds   "/bin/bash -c 'sourc…"   About a minute ago   Restarting (126) 12 seconds ago             hsds_sn_1
79a41bd29910   hdfgroup/hsds   "/bin/bash -c 'sourc…"   About a minute ago   Restarting (126) 21 seconds ago             hsds_head_1
>docker logs hsds_sn_1
/bin/bash: /entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/bin/bash: /entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/bin/bash: /entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/bin/bash: /entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/bin/bash: /entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/bin/bash: /entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/bin/bash: /entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/bin/bash: /entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/bin/bash: /entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
/bin/bash: /entrypoint.sh: /bin/bash^M: bad interpreter: No such file or directory
jreadey commented 2 years ago

Strange!
What do you get with: $ docker images hdfgroup/hsds?

anotherkhan commented 2 years ago

Please see below:

$ docker images hdfgroup/hsds
REPOSITORY      TAG       IMAGE ID       CREATED        SIZE
hdfgroup/hsds   latest    12a310c694d8   32 hours ago   486MB
jreadey commented 2 years ago

Looks like you built the image locally - likely building on Windows introduced some unexpected weirdness.

Try fetching the image from DockerHub. Do this:

$ ./stopall.sh
$ docker rmi 12a310c694d8
$ ./runall.sh

This should fetch the image from the hsds repo: https://hub.docker.com/repository/docker/hdfgroup/hsds/general.

jreadey commented 2 years ago

I've made some updates in the nodocker branch that should allow HSDS to run without Docker Desktop. Here's how to try it out:

  1. Run: git pull
  2. Run: git checkout nodocker
  3. From a python 3.x environment, run: python uninstall hsds
  4. From a python 3.x environment, run: python setup.py install
  5. Create a directory "data" if you don't have one
  6. Create a directory "data\hsdstest" if you don't have one
  7. From a python 3.x environment, run: runall.bat
  8. After a bit, you should see: "READY! use endpoint: http://localhost:5101"

From here you can follow the regular post setup instructions. To kill the server, do a ctrl-C and say "Y" and answer "Y" to "Terminate batch job?"

Will be most curious if anyone would like to report how this works for them. Since at least the version of windows I have doesn't support Unix Sockets (but see: https://bugs.python.org/issue33408), the runall.bat script runs HSDS without docker but using TCP/IP connections. This should work on MacOS/Linux too for anyone who wants to support cross-machine communication, but doesn't want to run Docker.

bilalshaikh42 commented 2 years ago

BTW, I'm curious if anyone has tried running HSDS on windows using WSL2 (windows subsystem for Linux). By all accounts (see https://medium.com/swlh/data-science-using-wsl2-seamless-integration-of-ubuntu-and-windows-10-dbe94cbac2b6) its very convenient for running Linux apps on Windows. I expect the --no-docker option would "just work" on WSL2, but don't have a way to test it myself.

I have! It works fine. Have not done much with it, but I do remember I got it to work and was able to use hsload to interact with it.

anotherkhan commented 2 years ago

Thank you @jreadey for making the update. I followed your instruction above. The server starts but shuts down immediately with following error.

(base) C:\Users\sami.a.khan\hsds>hsds --root_dir data --host localhost --port 5101 --password_file admin/config/passwd.txt --logfile hs.log   --loglevel DEBUG  --config_dir=admin/config --count=4
set logging to: 10
logfile: C:\Users\sami.a.khan\hsds\hs.log
DEBUG:root:looking for hsds-servicenode in PATH env var folders
INFO:root:using cmd_dir: C:\Users\sami.a.khan\Miniconda3\Scripts
endpoint: http://localhost:5101
using cmd_path: C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 23, in <module>
    from . import multiarray
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\multiarray.py", line 10, in <module>
    from . import overrides
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
    return ep.load()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
    return self.resolve()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 19, in <module>
    from .util.lruCache import LruCache
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\lruCache.py", line 12, in <module>
    import numpy
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\__init__.py", line 144, in <module>
    from . import core
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "C:\Users\sami.a.khan\Miniconda3\python.exe"
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 23, in <module>
    from . import multiarray
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\multiarray.py", line 10, in <module>
    from . import overrides
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
    return ep.load()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
    return self.resolve()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 19, in <module>
    from .util.lruCache import LruCache
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\lruCache.py", line 12, in <module>
    import numpy
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\__init__.py", line 144, in <module>
    from . import core
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "C:\Users\sami.a.khan\Miniconda3\python.exe"
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 23, in <module>
    from . import multiarray
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\multiarray.py", line 10, in <module>
    from . import overrides
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
    return ep.load()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
    return self.resolve()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 19, in <module>
    from .util.lruCache import LruCache
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\lruCache.py", line 12, in <module>
    import numpy
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\__init__.py", line 144, in <module>
    from . import core
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "C:\Users\sami.a.khan\Miniconda3\python.exe"
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 23, in <module>
    from . import multiarray
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\multiarray.py", line 10, in <module>
    from . import overrides
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
    return ep.load()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
    return self.resolve()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 19, in <module>
    from .util.lruCache import LruCache
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\lruCache.py", line 12, in <module>
    import numpy
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\__init__.py", line 144, in <module>
    from . import core
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "C:\Users\sami.a.khan\Miniconda3\python.exe"
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 23, in <module>
    from . import multiarray
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\multiarray.py", line 10, in <module>
    from . import overrides
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
    return ep.load()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
    return self.resolve()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 19, in <module>
    from .util.lruCache import LruCache
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\lruCache.py", line 12, in <module>
    import numpy
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\__init__.py", line 144, in <module>
    from . import core
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "C:\Users\sami.a.khan\Miniconda3\python.exe"
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 23, in <module>
    from . import multiarray
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\multiarray.py", line 10, in <module>
    from . import overrides
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
    return ep.load()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
    return self.resolve()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 19, in <module>
    from .util.lruCache import LruCache
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\lruCache.py", line 12, in <module>
    import numpy
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\__init__.py", line 144, in <module>
    from . import core
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numpy\core\__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "C:\Users\sami.a.khan\Miniconda3\python.exe"
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
INFO:root:all processes ready!
INFO:root:Ready after: 5.09 s
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
Exception in thread Thread-4:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

Exception in thread Thread-5:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)

WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)

Exception in thread Thread-6:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
READY! use endpoint: http://localhost:5101

got exception: I/O operation on closed file.
shutting down server
INFO:root:hsds app stop at 1648241168.5733004
DEBUG:root:all subprocesses exited
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\app.py", line 303, in main
    app.stop()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 364, in stop
    logging.debug("all subprocesses exited")
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 2002, in debug
    root.debug(msg, *args, **kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1366, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1514, in _log
    self.handle(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1524, in handle
    self.callHandlers(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1586, in callHandlers
    hdlr.handle(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 894, in handle
    self.emit(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1028, in emit
    stream.write(msg + self.terminator)
KeyboardInterrupt
INFO:root:hsds app stop at 1648241168.6051414
DEBUG:root:all subprocesses exited
jreadey commented 2 years ago

Sounds like something is out of whack with your numpy setup. Try this:

$ pip uninstall numpy
$ conda install numpy
anotherkhan commented 2 years ago

@jreadey, Thank you for quick reply. I update numpy as mentioned above. no more numpy related errors but server starts and shuts down.

>runall.bat

(base) C:\Users\sami.a.khan\hsds>hsds --root_dir data --host localhost --port 5101 --password_file admin/config/passwd.txt --logfile hs.log   --loglevel DEBUG  --config_dir=admin/config --count=4
set logging to: 10
logfile: C:\Users\sami.a.khan\hsds\hs.log
DEBUG:root:looking for hsds-servicenode in PATH env var folders
INFO:root:using cmd_dir: C:\Users\sami.a.khan\Miniconda3\Scripts
endpoint: http://localhost:5101
using cmd_path: C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
INFO:root:all processes ready!
INFO:root:Ready after: 5.05 s

READY! use endpoint: http://localhost:5101

Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)Traceback (most recent call last):

  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
        return ep.load()  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
    return get_distribution(dist).load_entry_point(group, name)

    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
return ep.load()
        return get_distribution(dist).load_entry_point(group, name)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
    return self.resolve()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
        return ep.load()
    return ep.load()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
return get_distribution(dist).load_entry_point(group, name)
return self.resolve()  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point

module = __import__(self.module_name, fromlist=['__name__'], level=0)  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load

      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    return self.resolve()    return self.resolve()

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
return ep.load()  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    return ep.load()  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve

    module = __import__(self.module_name, fromlist=['__name__'], level=0)
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    return self.resolve()module = __import__(self.module_name, fromlist=['__name__'], level=0)

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 24, in <module>
    return self.resolve()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    from .basenode import healthCheck,  baseInit
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
from . import servicenode  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\basenode.py", line 21, in <module>

    from . import servicenode
from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 24, in <module>
module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 24, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 24, in <module>
    module = __import__(self.module_name, fromlist=['__name__'], level=0)  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
import psutil
        from .basenode import healthCheck,  baseInitfrom .basenode import healthCheck,  baseInit
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
from .basenode import healthCheck,  baseInit

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\basenode.py", line 21, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\basenode.py", line 21, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\__init__.py", line 109, in <module>
            import psutil
from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 24, in <module>
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\__init__.py", line 109, in <module>
    from . import servicenode  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\basenode.py", line 21, in <module>
import psutil    from . import _pswindows as _psplatform

from .basenode import healthCheck,  baseInit
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 24, in <module>
    from . import _pswindows as _psplatform  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\__init__.py", line 109, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\_pswindows.py", line 35, in <module>
import psutil
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\basenode.py", line 21, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\_pswindows.py", line 35, in <module>
    from .basenode import healthCheck,  baseInitfrom . import _pswindows as _psplatform  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\__init__.py", line 109, in <module>
    from ._psutil_windows import ABOVE_NORMAL_PRIORITY_CLASS

    import psutil
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\basenode.py", line 21, in <module>
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\_pswindows.py", line 35, in <module>
from ._psutil_windows import ABOVE_NORMAL_PRIORITY_CLASS
ModuleNotFoundErrorfrom . import _pswindows as _psplatform
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\__init__.py", line 109, in <module>
    :
import psutilModuleNotFoundErrorfrom ._psutil_windows import ABOVE_NORMAL_PRIORITY_CLASSNo module named 'psutil._psutil_windows'  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\_pswindows.py", line 35, in <module>
    from . import _pswindows as _psplatform
:

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\__init__.py", line 109, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\_pswindows.py", line 35, in <module>
No module named 'psutil._psutil_windows'from ._psutil_windows import ABOVE_NORMAL_PRIORITY_CLASS

ModuleNotFoundError        ModuleNotFoundError: from . import _pswindows as _psplatformfrom ._psutil_windows import ABOVE_NORMAL_PRIORITY_CLASS: No module named 'psutil._psutil_windows'

No module named 'psutil._psutil_windows'
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\psutil\_pswindows.py", line 35, in <module>
ModuleNotFoundError
    : from ._psutil_windows import ABOVE_NORMAL_PRIORITY_CLASSNo module named 'psutil._psutil_windows'

ModuleNotFoundError: No module named 'psutil._psutil_windows'
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

Exception in thread Thread-4:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
Exception in thread Thread-6:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

got exception: I/O operation on closed file.
shutting down server
INFO:root:hsds app stop at 1648242296.7249045
DEBUG:root:all subprocesses exited
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\app.py", line 303, in main
    app.stop()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 364, in stop
    logging.debug("all subprocesses exited")
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 2002, in debug
    root.debug(msg, *args, **kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1366, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1514, in _log
    self.handle(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1524, in handle
    self.callHandlers(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1586, in callHandlers
    hdlr.handle(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 894, in handle
    self.emit(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1028, in emit
    stream.write(msg + self.terminator)
KeyboardInterrupt
INFO:root:hsds app stop at 1648242296.761941
DEBUG:root:all subprocesses exited
jreadey commented 2 years ago

Some progress!

Try the same trick with psutil:

$ pip uninstall psutil
$ conda install psutil
anotherkhan commented 2 years ago

Thanks :-) @jreadey. Do you recommend to reinstall miniconda?

runall.bat
(base) C:\Users\sami.a.khan\hsds>hsds --root_dir data --host localhost --port 5101 --password_file admin/config/passwd.txt --logfile hs.log   --loglevel DEBUG  --config_dir=admin/config --count=4
set logging to: 10
logfile: C:\Users\sami.a.khan\hsds\hs.log
DEBUG:root:looking for hsds-servicenode in PATH env var folders
INFO:root:using cmd_dir: C:\Users\sami.a.khan\Miniconda3\Scripts
endpoint: http://localhost:5101
using cmd_path: C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
Traceback (most recent call last):
      File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())            sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point

        return get_distribution(dist).load_entry_point(group, name)      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)return get_distribution(dist).load_entry_point(group, name)
return get_distribution(dist).load_entry_point(group, name)return get_distribution(dist).load_entry_point(group, name)

      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point

                return ep.load()  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
return ep.load()return ep.load()return ep.load()return ep.load()

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
return ep.load()
return self.resolve()return self.resolve()return self.resolve()  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load

    return self.resolve()  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
return self.resolve()

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    return self.resolve()module = __import__(self.module_name, fromlist=['__name__'], level=0)  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)

module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    module = __import__(self.module_name, fromlist=['__name__'], level=0)  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
        module = __import__(self.module_name, fromlist=['__name__'], level=0)
    from . import servicenodemodule = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    from . import servicenode

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>
    from . import servicenode      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>
        from . import servicenode
from .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domainsfrom . import servicenodefrom .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domains
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>
    from .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domains

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>
        from .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domains        from .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domainsfrom .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domains
from .util.storUtil import getStorKeys, getCompressorsfrom .util.storUtil import getStorKeys, getCompressors

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>
from .util.storUtil import getStorKeys, getCompressors  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>

            from .util.storUtil import getStorKeys, getCompressors  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>
    from .util.storUtil import getStorKeys, getCompressorsfrom .util.storUtil import getStorKeys, getCompressorsimport numcodecs as codecs
import numcodecs as codecs

      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>
import numcodecs as codecs  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>

            import numcodecs as codecsimport numcodecs as codecs  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>
import numcodecs as codecsfrom numcodecs.shuffle import Shufflefrom numcodecs.shuffle import Shuffle

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>
from numcodecs.shuffle import Shuffle      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>
from numcodecs.shuffle import Shuffle
from numcodecs.shuffle import Shuffle
from numcodecs.shuffle import Shuffle  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>

from ._shuffle import _doShuffle, _doUnshuffle  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>

from ._shuffle import _doShuffle, _doUnshuffle  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>

from ._shuffle import _doShuffle, _doUnshuffleModuleNotFoundError    from ._shuffle import _doShuffle, _doUnshuffleModuleNotFoundError
: from ._shuffle import _doShuffle, _doUnshuffle
: from ._shuffle import _doShuffle, _doUnshuffle
No module named 'numcodecs._shuffle'
ModuleNotFoundErrorNo module named 'numcodecs._shuffle'ModuleNotFoundErrorModuleNotFoundError
ModuleNotFoundError:
: : : No module named 'numcodecs._shuffle'No module named 'numcodecs._shuffle'No module named 'numcodecs._shuffle'No module named 'numcodecs._shuffle'

DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
INFO:root:all processes ready!
INFO:root:Ready after: 5.09 s
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)

WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
Exception in thread Thread-4:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

Exception in thread Thread-6:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
READY! use endpoint: http://localhost:5101

got exception: I/O operation on closed file.
shutting down server
INFO:root:hsds app stop at 1648243049.8040502
DEBUG:root:all subprocesses exited
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\app.py", line 303, in main
    app.stop()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 364, in stop
    logging.debug("all subprocesses exited")
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 2002, in debug
    root.debug(msg, *args, **kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1366, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1514, in _log
    self.handle(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1524, in handle
    self.callHandlers(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1586, in callHandlers
    hdlr.handle(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 894, in handle
    self.emit(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1028, in emit
    stream.write(msg + self.terminator)
KeyboardInterrupt
INFO:root:hsds app stop at 1648243049.8315725
(base) C:\Users\sami.a.khan\hsds>hsds --root_dir data --host localhost --port 5101 --password_file admin/config/passwd.txt --logfile hs.log   --loglevel DEBUG  --config_dir=admin/config --count=4
set logging to: 10
logfile: C:\Users\sami.a.khan\hsds\hs.log
DEBUG:root:looking for hsds-servicenode in PATH env var folders
INFO:root:using cmd_dir: C:\Users\sami.a.khan\Miniconda3\Scripts
endpoint: http://localhost:5101
using cmd_path: C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
Traceback (most recent call last):
      File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-node-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())            sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds-node')())  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point

        return get_distribution(dist).load_entry_point(group, name)      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)return get_distribution(dist).load_entry_point(group, name)
return get_distribution(dist).load_entry_point(group, name)return get_distribution(dist).load_entry_point(group, name)

      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point

                return ep.load()  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2846, in load_entry_point
return ep.load()return ep.load()return ep.load()return ep.load()

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load
return ep.load()
return self.resolve()return self.resolve()return self.resolve()  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2450, in load

    return self.resolve()  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
return self.resolve()

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    return self.resolve()module = __import__(self.module_name, fromlist=['__name__'], level=0)  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)

module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\pkg_resources\__init__.py", line 2456, in resolve
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    module = __import__(self.module_name, fromlist=['__name__'], level=0)  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
        module = __import__(self.module_name, fromlist=['__name__'], level=0)
    from . import servicenodemodule = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
    from . import servicenode

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>
from . import servicenode
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\node_runner.py", line 5, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>
    from . import servicenode      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>
        from . import servicenode
from .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domainsfrom . import servicenodefrom .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domains
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>
    from .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domains

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\servicenode.py", line 27, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>
        from .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domains        from .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domainsfrom .domain_sn import GET_Domain, PUT_Domain, DELETE_Domain, GET_Domains
from .util.storUtil import getStorKeys, getCompressorsfrom .util.storUtil import getStorKeys, getCompressors

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>
from .util.storUtil import getStorKeys, getCompressors  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\domain_sn.py", line 39, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>

            from .util.storUtil import getStorKeys, getCompressors  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>
    from .util.storUtil import getStorKeys, getCompressorsfrom .util.storUtil import getStorKeys, getCompressorsimport numcodecs as codecs
import numcodecs as codecs

      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\util\storUtil.py", line 20, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>
import numcodecs as codecs  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>

            import numcodecs as codecsimport numcodecs as codecs  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>
import numcodecs as codecsfrom numcodecs.shuffle import Shufflefrom numcodecs.shuffle import Shuffle

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\__init__.py", line 100, in <module>
      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>
from numcodecs.shuffle import Shuffle      File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>
from numcodecs.shuffle import Shuffle
from numcodecs.shuffle import Shuffle
from numcodecs.shuffle import Shuffle  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>

from ._shuffle import _doShuffle, _doUnshuffle  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>

from ._shuffle import _doShuffle, _doUnshuffle  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>

  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\numcodecs\shuffle.py", line 4, in <module>

from ._shuffle import _doShuffle, _doUnshuffleModuleNotFoundError    from ._shuffle import _doShuffle, _doUnshuffleModuleNotFoundError
: from ._shuffle import _doShuffle, _doUnshuffle
: from ._shuffle import _doShuffle, _doUnshuffle
No module named 'numcodecs._shuffle'
ModuleNotFoundErrorNo module named 'numcodecs._shuffle'ModuleNotFoundErrorModuleNotFoundError
ModuleNotFoundError:
: : : No module named 'numcodecs._shuffle'No module named 'numcodecs._shuffle'No module named 'numcodecs._shuffle'No module named 'numcodecs._shuffle'

DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
DEBUG:root:0/6 ready
DEBUG:root:sleeping for 1
INFO:root:all processes ready!
INFO:root:Ready after: 5.09 s
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)

WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
Exception in thread Thread-4:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
WARNING:root:process C:\Users\sami.a.khan\Miniconda3\python.exe ended, result: ('', None)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file

Exception in thread Thread-6:
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\sami.a.khan\Miniconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 15, in _enqueue_output
    for line in iter(out.readline, b''):
ValueError: I/O operation on closed file
READY! use endpoint: http://localhost:5101

got exception: I/O operation on closed file.
shutting down server
INFO:root:hsds app stop at 1648243049.8040502
DEBUG:root:all subprocesses exited
Traceback (most recent call last):
  File "C:\Users\sami.a.khan\Miniconda3\Scripts\hsds-script.py", line 33, in <module>
    sys.exit(load_entry_point('hsds==0.7.0b11', 'console_scripts', 'hsds')())
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\app.py", line 303, in main
    app.stop()
  File "C:\Users\sami.a.khan\Miniconda3\lib\site-packages\hsds-0.7.0b11-py3.7.egg\hsds\hsds_app.py", line 364, in stop
    logging.debug("all subprocesses exited")
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 2002, in debug
    root.debug(msg, *args, **kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1366, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1514, in _log
    self.handle(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1524, in handle
    self.callHandlers(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1586, in callHandlers
    hdlr.handle(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 894, in handle
    self.emit(record)
  File "C:\Users\sami.a.khan\Miniconda3\lib\logging\__init__.py", line 1028, in emit
    stream.write(msg + self.terminator)
KeyboardInterrupt
INFO:root:hsds app stop at 1648243049.8315725
jreadey commented 2 years ago

Actually, I get the same error you are seeing when I try running setup.py in a clean Anaconda environment. Looks like newer version of some packages has introduced incompatibilities.

Let me take a look at this and see if I can fix up the install script.

jreadey commented 2 years ago

It seems the newer version of aiobotocore is the culprit. I've updated the setup.py script to use aiobotcore==2.1.0. This is working for me with both MacOS and Windows. Please give it a try and let me know how it goes.

jreadey commented 2 years ago

Reminder: to run on Windows, use the runall.bat script.

jreadey commented 1 year ago

Closing this issue even though I'm not 100% on HSDS running in Windows (and have no means to verify).

In general, I'd recommend using WSL2 on Windows as offering the best approach (as @bilalshaikh42 verified).

jreadey commented 1 year ago

Just put in some fixes for HSDS on Windows - it should be running correctly now. Docker on Windows is not supported, but you "runall.bat" (i.e. running the hsds app) works. The github python-package workflow runs on Linux Ubuntu, Mac, and Windows-latest, so should catch any future regressions.