Closed cdhainaut closed 3 years ago
Hey @carlitador,
A good option for your software installations is to install the software on a shared FSx. You can use the import_path
configuration to initialize the FSx with your software package from an S3 bucket. Please take a look at the FSx configuration option [fsx] section of the documentation for further information.
Example Configuration taken from the HPC Workshop Section "Building a High-Performance File System":
aws_region_name = ${REGION}
[global]
cluster_template = default
update_check = false
sanity_check = false
[cluster default]
key_name = lab-4-your-key
vpc_settings = public
base_os = alinux2
ebs_settings = myebs
fsx_settings = myfsx
compute_instance_type = c5.18xlarge
master_instance_type = c5.xlarge
cluster_type = ondemand
placement_group = DYNAMIC
placement = compute
max_queue_size = 8
initial_queue_size = 0
disable_hyperthreading = true
scheduler = slurm
[vpc public]
vpc_id = ${VPC_ID}
master_subnet_id = ${SUBNET_ID}
[ebs myebs]
shared_dir = /shared
volume_type = gp2
volume_size = 20
[fsx myfsx]
shared_dir = /lustre
storage_capacity = 1200
import_path = s3://mybucket-${BUCKET_POSTFIX} # add or application to this bucket
deployment_type = SCRATCH_2
[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}
Thanks @BennX, this is indeed the best approach.
Hi @carlitador, for Q.2, not sure I got what you mean. CLI loglevel is INFO for stdout and DEBUG for log file (~/.parallelcluster/pcluster-cli.log
) and cannot be changed.
Hi guys, Thanks for your answers. I will have a look at the proposed setup with FSx. For the debug messages, why I have all the logs inside the CLI then ? (See picture below...)
Thanks
What is the pcluster version that you running? Did you install it in a clean environment, like for example a virtualenv?
I dont use a virtual environnment. I installed it via pip install. pcluster version:
pcluster version
DEBUG:pcluster.cli:pcluster CLI starting
DEBUG:pcluster.cli:Namespace(command='version', func=<function version at 0x7fbad9731280>)
2.10.0
Hello, Any idea how I can remove these "DEBUG" messages? Thanks
Hi @carlitador,
if you install ParallelCluster in a virtualenv and you run it from there, do you get the same loglevel?
Can you please check if there is something regarding loglevel set in your environment, looking at the output of env | sort
command?
Hi @lukeseawalker,
I tried env | sort
, do you think it could be related to these entries ?
GJS_DEBUG_OUTPUT=stderr GJS_DEBUG_TOPICS=JS ERROR;JS LOG
Here is the rest of the command output for information:
_CE_CONDA= _CE_M= CINNAMON_VERSION=4.4.8 COLORTERM=truecolor CONDA_DEFAULT_ENV=default CONDA_EXE=/home/charles/miniconda3/bin/conda CONDA_PREFIX_1=/home/charles/miniconda3 CONDA_PREFIX=/home/charles/miniconda3/envs/default CONDA_PROMPT_MODIFIER=(default) CONDA_PYTHON_EXE=/home/charles/miniconda3/bin/python CONDA_SHLVL=2 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus DEFAULTS_PATH=/usr/share/gconf/cinnamon.default.path DESKTOP_SESSION=cinnamon DISPLAY=:0 GDM_LANG=fr_FR GDMSESSION=cinnamon GJS_DEBUG_OUTPUT=stderr GJS_DEBUG_TOPICS=JS ERROR;JS LOG GNOME_DESKTOP_SESSION_ID=this-is-deprecated GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/abdc9050_283a_4c19_be38_469faeb551b2 GNOME_TERMINAL_SERVICE=:1.82 GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1 GTK_MODULES=gail:atk-bridge GTK_OVERLAY_SCROLLING=1 HOME=/home/charles LANG=fr_FR.UTF-8 LANGUAGE=fr_FR LC_NUMERIC=en_US.UTF-8 LESSCLOSE=/usr/bin/lesspipe %s %s LESSOPEN=| /usr/bin/lesspipe %s LOGNAME=charles LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: MANDATORY_PATH=/usr/share/gconf/cinnamon.mandatory.path OLDPWD=/home/charles PATH=/home/charles/miniconda3/envs/default/bin:/home/charles/miniconda3/condabin:/opt/CD-adapco/12.06.011/STAR-View+12.06.011/bin:/opt/CD-adapco/12.06.011/STAR-CCM+12.06.011/star/bin:/home/charles/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin PWD=/mnt/sda2/Work/CDYYY-Dev/python-utils PYTHONPATH=:/mnt/sda2/Work/CDYYY-Dev/python-utils PYUTILS=/mnt/sda2/Work/CDYYY-Dev/python-utils QT_ACCESSIBILITY=1 QT_QPA_PLATFORMTHEME=qt5ct SESSION_MANAGER=local/charles-ThinkPad-E15:@/tmp/.ICE-unix/2072,unix/charles-ThinkPad-E15:/tmp/.ICE-unix/2072 SHELL=/bin/bash SHLVL=1 SSH_AGENT_PID=2140 SSH_AUTH_SOCK=/run/user/1000/keyring/ssh TERM=xterm-256color USER=charles _=/usr/bin/env VTE_VERSION=5202 WORK_PATH=/mnt/sda2/Work XAUTHORITY=/home/charles/.Xauthority XDG_CONFIG_DIRS=/etc/xdg/xdg-cinnamon:/etc/xdg XDG_CURRENT_DESKTOP=X-Cinnamon XDG_DATA_DIRS=/usr/share/cinnamon:/usr/share/gnome:/home/charles/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share:/var/lib/snapd/desktop XDG_GREETER_DATA_DIR=/var/lib/lightdm-data/charles XDG_RUNTIME_DIR=/run/user/1000 XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0 XDG_SEAT=seat0 XDG_SESSION_DESKTOP=cinnamon XDG_SESSION_ID=c2 XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0 XDG_SESSION_TYPE=x11 XDG_VTNR=7
Thanks
Can you please post the output of the following commands:
conda info
conda list
conda config --show
Hi @lukeseawalker, there we go:
conda info
active environment : default
active env location : /home/charles/miniconda3/envs/default
shell level : 2
user config file : /home/charles/.condarc
populated config files :
conda version : 4.8.3
conda-build version : not installed
python version : 3.7.7.final.0
virtual packages : __glibc=2.27
base environment : /home/charles/miniconda3 (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/charles/miniconda3/pkgs
/home/charles/.conda/pkgs
envs directories : /home/charles/miniconda3/envs
/home/charles/.conda/envs
platform : linux-64
user-agent : conda/4.8.3 requests/2.23.0 CPython/3.7.7 Linux/5.4.0-54-generic linuxmint/19.3 glibc/2.27
UID:GID : 1000:1000
netrc file : None
offline mode : False
conda list
# packages in environment at /home/charles/miniconda3/envs/default:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
aerosandbox 2.2.11 pypi_0 pypi
airfoil-db 1.4.3 pypi_0 pypi
alabaster 0.7.12 py_0
appdirs 1.4.4 py_0
asgiref 3.3.1 pyhd3eb1b0_0
astroid 2.4.2 py38_0
attrs 19.3.0 pypi_0 pypi
aws-parallelcluster 2.10.0 pypi_0 pypi
babel 2.8.0 py_0
backcall 0.2.0 py_0
beautifulsoup4 4.9.1 py38_0
black 19.10b0 py_0
blas 1.0 mkl
boto3 1.16.24 pypi_0 pypi
botocore 1.19.24 pypi_0 pypi
brotlipy 0.7.0 py38h7b6447c_1000
bzip2 1.0.8 h7b6447c_0
ca-certificates 2020.10.14 0
casadi 3.5.5 pypi_0 pypi
certifi 2020.11.8 py38h06a4308_0
cffi 1.14.3 py38he30daa8_0
chardet 3.0.4 py38_1003
click 7.1.2 py_0
conda 4.9.1 py38h06a4308_0
conda-build 3.20.2 py38_0
conda-package-handling 1.6.1 py38h7b6447c_0
cryptography 3.1 py38h1ba5d50_0
cycler 0.10.0 py38_0
cython 0.29.14 pypi_0 pypi
dash 1.13.4 pyh9f0ad1d_0 conda-forge
dash-bootstrap-components 0.10.3 pyh9f0ad1d_0 conda-forge
dash-core-components 1.10.1 pyh9f0ad1d_0 conda-forge
dash-html-components 1.0.3 pyh9f0ad1d_0 conda-forge
dash-renderer 1.5.1 pyh9f0ad1d_0 conda-forge
dash-table 4.8.1 pyh9f0ad1d_0 conda-forge
dataclasses 0.6 pypi_0 pypi
dbus 1.13.16 hb2f20db_0
decorator 4.4.2 py_0
dill 0.3.2 pypi_0 pypi
django 3.1.3 pyhd3eb1b0_0
docutils 0.15.2 py38_0
expat 2.2.9 he6710b0_2
filelock 3.0.12 py_0
flask 1.1.2 py_0
flask-compress 1.5.0 py_0
fontconfig 2.13.1 he4413a7_1000 conda-forge
freetype 2.10.2 h5ab3b9f_0
future 0.18.2 py38_1
glib 2.65.0 h3eb4bd4_0
glob2 0.7 py_0
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb31296c_0
icu 58.2 he6710b0_3
idna 2.10 py_0
imagesize 1.2.0 py_0
intel-openmp 2020.2 254
ipaddress 1.0.23 pypi_0 pypi
ipykernel 5.3.4 py38h5ca1d4c_0
ipython 7.19.0 py38hb070fc8_0
ipython_genutils 0.2.0 py38_0
isort 5.6.4 py_0
itsdangerous 1.1.0 py_0
jedi 0.17.2 py38_0
jinja2 2.11.2 py_0
jmespath 0.10.0 py_0
joblib 0.16.0 py_0
jpeg 9b h024ee3a_2
jupyter_client 6.1.7 py_0
jupyter_core 4.6.3 py38_0
katrain 1.3.4 pypi_0 pypi
kivy 2.0.0rc2 pypi_0 pypi
kivy-garden 0.1.4 pypi_0 pypi
kivymd 0.104.1 pypi_0 pypi
kiwisolver 1.2.0 py38hfd86e86_0
krb5 1.17.1 h173b8e3_0
lazy-object-proxy 1.4.3 py38h7b6447c_0
ld_impl_linux-64 2.33.1 h53a641e_7
libarchive 3.4.2 h62408e4_0
libedit 3.1.20191231 h14c3975_1
libffi 3.3 he6710b0_2
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
liblief 0.10.1 he6710b0_0
libpng 1.6.37 hbc83047_0
libpq 12.2 h20c2e04_0
libsodium 1.0.18 h7b6447c_0
libstdcxx-ng 9.1.0 hdf63c60_0
libuuid 2.32.1 h14c3975_1000 conda-forge
libxcb 1.14 h7b6447c_0
libxml2 2.9.10 he19cac6_1
lz4-c 1.9.2 he6710b0_1
machupx 2.4.1 pypi_0 pypi
markupsafe 1.1.1 py38h7b6447c_0
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py38hef1b27d_0
mccabe 0.6.1 py38_1
mkl 2020.2 256
mkl-service 2.3.0 py38he904b0f_0
mkl_fft 1.2.0 py38h23d657b_0
mkl_random 1.1.1 py38h0573a6f_0
more-itertools 8.4.0 pypi_0 pypi
multiprocess 0.70.10 pypi_0 pypi
mypy_extensions 0.4.3 py38_0
ncurses 6.2 he6710b0_1
nose 1.3.7 py38_2
numpy 1.19.1 py38hbc911f0_0
numpy-base 1.19.1 py38hfa32c7d_0
numpy-stl 2.11.2 py38h8790de6_0 conda-forge
occt 7.4.0 h9121d39_3 conda-forge
openssl 1.1.1h h7b6447c_0
packaging 20.4 py_0
pandas 1.0.5 py38h0573a6f_0
parso 0.7.0 py_0
patchelf 0.11 he6710b0_0
pathspec 0.7.0 py_0
pcre 8.44 he6710b0_0
pexpect 4.8.0 py38_0
pgi 0.0.11.2 pypi_0 pypi
pickleshare 0.7.5 py38_1000
pillow 7.2.0 pypi_0 pypi
pip 20.2.2 py38_0
pkginfo 1.5.0.1 py38_0
plotly 4.8.2 py_0
pluggy 0.13.1 pypi_0 pypi
prompt-toolkit 3.0.8 py_0
psutil 5.7.2 py38h7b6447c_0
psycopg2 2.8.5 py38h1ba5d50_0
ptyprocess 0.6.0 py38_0
py 1.9.0 pypi_0 pypi
py-lief 0.10.1 py38h403a769_0
pycairo 1.19.1 pypi_0 pypi
pycosat 0.6.3 py38h7b6447c_1
pycparser 2.20 py_2
pygments 2.7.1 py_0
pygobject 3.36.1 pypi_0 pypi
pylint 2.6.0 py38_0
pyopenssl 19.1.0 py_1
pyparsing 2.4.7 py_0
pyqt 5.9.2 py38h05f1152_4
pysocks 1.7.1 py38_0
pytest 5.4.3 pypi_0 pypi
python 3.8.5 h7579374_1
python-dateutil 2.8.1 py_0
python-libarchive-c 2.9 py_0
python-utils 2.4.0 py38_0
python_abi 3.8 1_cp38 conda-forge
pythonocc-core 7.4.0 py38h8e684b0_0 conda-forge
pytz 2020.1 py_0
pyyaml 5.3.1 py38h7b6447c_1
pyzmq 19.0.2 py38he6710b0_1
qt 5.9.7 h5867ecd_1
readline 8.0 h7b6447c_0
regex 2020.10.15 py38h7b6447c_0
requests 2.24.0 py_0
retrying 1.3.3 py_2
ripgrep 11.0.2 he32d670_0
ruamel-yaml 0.16.12 pypi_0 pypi
ruamel-yaml-clib 0.2.2 pypi_0 pypi
ruamel_yaml 0.15.87 py38h7b6447c_1
s3transfer 0.3.3 py38_0
scikit-learn 0.23.1 py38h423224d_0
scipy 1.5.2 py38h0b6359f_0
screeninfo 0.6.5 pypi_0 pypi
seaborn 0.11.0 pypi_0 pypi
selenium 3.141.0 py38h7b6447c_1000
setproctitle 1.1.10 pypi_0 pypi
setuptools 49.6.0 py38_0
sip 4.19.13 py38he6710b0_0
six 1.15.0 py_0
snowballstemmer 2.0.0 py_0
soupsieve 2.0.1 py_0
sphinx 3.2.1 py_0
sphinx_rtd_theme 0.4.3 py_0
sphinxcontrib-applehelp 1.0.2 py_0
sphinxcontrib-devhelp 1.0.2 py_0
sphinxcontrib-htmlhelp 1.0.3 py_0
sphinxcontrib-jsmath 1.0.1 py_0
sphinxcontrib-qthelp 1.0.3 py_0
sphinxcontrib-serializinghtml 1.1.4 py_0
sqlite 3.33.0 h62c20be_0
sqlparse 0.4.1 py_0
tabulate 0.8.7 pypi_0 pypi
tbb 2020.2 hc9558a2_0 conda-forge
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 hbc83047_0
toml 0.10.1 py_0
tornado 6.0.4 py38h7b6447c_1
tqdm 4.49.0 py_0
traitlets 5.0.5 py_0
typed-ast 1.4.1 py38h7b6447c_0
typing_extensions 3.7.4.3 py_0
urllib3 1.25.10 py_0
vext 0.7.4 pypi_0 pypi
vext-gi 0.7.4 pypi_0 pypi
wcwidth 0.2.5 py_0
werkzeug 1.0.1 py_0
wheel 0.35.1 py_0
wrapt 1.11.2 py38h7b6447c_0
xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.10 h516909a_0 conda-forge
xorg-libsm 1.2.3 h84519dc_1000 conda-forge
xorg-libx11 1.6.12 h516909a_0 conda-forge
xorg-libxt 1.2.0 h516909a_0 conda-forge
xorg-xproto 7.0.31 h14c3975_1007 conda-forge
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
zeromq 4.3.3 he6710b0_3
zlib 1.2.11 h7b6447c_3
zstd 1.4.5 h9ceee32_0
conda config --show
add_anaconda_token: True
add_pip_as_python_dependency: True
aggressive_update_packages:
- ca-certificates
- certifi
- openssl
allow_conda_downgrades: False
allow_cycles: True
allow_non_channel_urls: False
allow_softlinks: False
always_copy: False
always_softlink: False
always_yes: None
anaconda_upload: None
auto_activate_base: True
auto_stack: 0
auto_update_conda: True
bld_path:
changeps1: True
channel_alias: https://conda.anaconda.org
channel_priority: flexible
channels:
- defaults
client_ssl_cert: None
client_ssl_cert_key: None
clobber: False
conda_build: {}
create_default_packages: []
croot: /home/charles/miniconda3/conda-bld
custom_channels:
pkgs/main: https://repo.anaconda.com
pkgs/r: https://repo.anaconda.com
pkgs/pro: https://repo.anaconda.com
custom_multichannels:
defaults:
- https://repo.anaconda.com/pkgs/main
- https://repo.anaconda.com/pkgs/r
local:
debug: False
default_channels:
- https://repo.anaconda.com/pkgs/main
- https://repo.anaconda.com/pkgs/r
default_python: 3.7
default_threads: None
deps_modifier: not_set
dev: False
disallowed_packages: []
download_only: False
dry_run: False
enable_private_envs: False
env_prompt: ({default_env})
envs_dirs:
- /home/charles/miniconda3/envs
- /home/charles/.conda/envs
error_upload_url: https://conda.io/conda-post/unexpected-error
execute_threads: 1
extra_safety_checks: False
force: False
force_32bit: False
force_reinstall: False
force_remove: False
ignore_pinned: False
json: False
local_repodata_ttl: 1
migrated_channel_aliases: []
migrated_custom_channels: {}
non_admin_enabled: True
notify_outdated_conda: True
offline: False
override_channels_enabled: True
path_conflict: clobber
pinned_packages: []
pip_interop_enabled: False
pkgs_dirs:
- /home/charles/miniconda3/pkgs
- /home/charles/.conda/pkgs
proxy_servers: {}
quiet: False
remote_backoff_factor: 1
remote_connect_timeout_secs: 9.15
remote_max_retries: 3
remote_read_timeout_secs: 60.0
repodata_fns:
- current_repodata.json
- repodata.json
repodata_threads: None
report_errors: None
restore_free_channel: False
rollback_enabled: True
root_prefix: /home/charles/miniconda3
safety_checks: warn
sat_solver: pycosat
separate_format_cache: False
shortcuts: True
show_channel_urls: None
solver_ignore_timestamps: False
ssl_verify: True
subdir: linux-64
subdirs:
- linux-64
- noarch
target_prefix_override:
track_features: []
unsatisfiable_hints: True
unsatisfiable_hints_check_depth: 2
update_modifier: update_specs
use_index_cache: False
use_local: False
use_only_tar_bz2: False
verbosity: 0
verify_threads: 1
whitelist_channels: []
Thanks for helping
Hello, Any clue please ? Thanks
Hi @carlitador ,
Sorry for the late reply. Both I and @lukeseawalker failed to reproduce your issue.
We were trying to see if there is any thing peculiar with your conda environment, but we can see that debug: False
and verbosity: 0
and could not find anything obvious that could be causing the issue.
Unfortunately, we don't have much to suggest beyond this point. It would be helpful if you can provide us with more detail on how to reproduce this issue. As a last resort, You might want to try to create a new, clean environment.
Best, Rex
This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.
Hello,
This isnt an issue but more a bunch of questions...
The documentation recommends not to use custom AMI, as it forces the user to create a new compatible AMI for every parallel cluster update. I don't really know the best way to avoid that, considering I need to install a software on each of my cluster nodes. Pulling the software from s3 would probably a quite slow solution because the software is quite heavy. What do you suggest would be good process to have robust setup with my software already installed on each nodes ? Do you think Docker containers could be a solution ?
Its been a long time since I didn't use parallel cluster. When I installed the new version, I had the "DEBUG" mode enabled by default. How can I disable it ?
Thank you Cheers