On-board Shengjie Xu - Githubissues

sonaalKant commented 7 months ago

Please follow this checklist. Post any errors you face in this issue.

[x] Acknowledge the pledge to put time in the project here
[x] Read How to organize your work
[x] Read Quick start for developing
[x] Make sure the Docker dev container works
[x] Make sure the unit tests run successfully
[x] Read Sorrentum Python coding style guide
[x] Fork, star, watch the Sorrentum repo so that GitHub promotes our repo (we gotta work the system)
[x] Get assigned a warm-up issue
[x] Learn about the Morning Email
[x] How to do a review
[ ] If you are graduating soon and you would like to get a full-time job in one of the companies in the Sorrentum ecosystem reach out to GP at gp@sorrentum.org

FYI @gpsaggese @samarth9008

jayhsu0627 commented 7 months ago

@sonaalKant Facing this error,

  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/Users/sheng***/src/venv/amp.client_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/sheng***/src/venv/amp.client_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/sheng***/src/venv/amp.client_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 550, in run
          self.add_defaults()
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 588, in add_defaults
          sdist.add_defaults(self)
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 102, in add_defaults
          super().add_defaults()
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/private/var/folders/v0/tbqkhzgx1132y1x64chm4yvw0000gn/T/pip-build-env-k1gdjdng/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[Process completed]

After investigating the commit history (commit 32d1598) and #437, fix this issue by edit [~/src/sorrentum1/dev_scripts/client_setup/requirements.txt](https://github.com/sorrentum/sorrentum/blob/master/dev_scripts/client_setup/requirements.txt) into the following code

pyyaml == 5.3.1 # Added this to support aws cli
awscli >= 1.19  # For aws CLI.
boto3 >= 1.20.17
# Keep in sync with `.github/gh_requirements.txt`, see CmTask6488.
docker < 7
docker-compose >= 1.29.0
invoke >= 1.5.0
jupyter  # For publish_notebook
jupyter_contrib_nbextensions
pandas   # For publish_notebook
poetry
pytest >= "6.0.0"
pytest-cov
pytest-instafail
pytest-xdist
python-dateutil
s3fs  # For tools like `publish_notebook.py`.
tqdm

output:

.
.
.
Successfully installed anyio-4.3.0 appnope-0.1.4 argon2-cffi-23.1.0 argon2-cffi-bindings-21.2.0 asttokens-2.4.1 attrs-23.2.0 awscli-1.32.47 bcrypt-4.1.2 beautifulsoup4-4.12.3 bleach-6.1.0 boto3-1.34.47 botocore-1.34.47 build-1.0.3 cachecontrol-0.13.1 certifi-2024.2.2 cffi-1.16.0 charset-normalizer-3.3.2 cleo-2.1.0 colorama-0.4.4 comm-0.2.1 coverage-7.4.2 crashtest-0.4.1 cryptography-42.0.4 debugpy-1.8.1 decorator-5.1.1 defusedxml-0.7.1 distlib-0.3.8 distro-1.9.0 docker-6.1.3 docker-compose-1.29.2 dockerpty-0.4.1 docopt-0.6.2 docutils-0.16 dulwich-0.21.7 entrypoints-0.4 execnet-2.0.2 executing-2.0.1 fastjsonschema-2.19.1 filelock-3.13.1 fsspec-2024.2.0 idna-3.6 importlib-metadata-7.0.1 iniconfig-2.0.0 installer-0.7.0 invoke-2.2.0 ipykernel-6.29.2 ipython-8.21.0 ipython-genutils-0.2.0 ipywidgets-8.1.2 jaraco.classes-3.3.1 jedi-0.19.1 jinja2-3.1.3 jmespath-1.0.1 jsonschema-3.2.0 jupyter-1.0.0 jupyter-client-7.4.9 jupyter-console-6.6.3 jupyter-contrib-core-0.4.2 jupyter-core-5.7.1 jupyter-events-0.6.3 jupyter-highlight-selected-word-0.2.0 jupyter-nbextensions-configurator-0.6.3 jupyter-server-2.10.0 jupyter-server-terminals-0.5.2 jupyter_contrib_nbextensions-0.7.0 jupyterlab-pygments-0.3.0 jupyterlab-widgets-3.0.10 keyring-24.3.0 lxml-5.1.0 markupsafe-2.1.5 matplotlib-inline-0.1.6 mistune-3.0.2 more-itertools-10.2.0 msgpack-1.0.7 nbclassic-1.0.0 nbclient-0.9.0 nbconvert-7.16.1 nbformat-5.9.2 nest-asyncio-1.6.0 notebook-6.5.6 notebook-shim-0.2.4 numpy-1.26.4 overrides-7.7.0 packaging-23.2 pandas-2.2.0 pandocfilters-1.5.1 paramiko-3.4.0 parso-0.8.3 pexpect-4.9.0 pkginfo-1.9.6 platformdirs-3.11.0 pluggy-1.4.0 poetry-1.7.1 poetry-core-1.8.1 poetry-plugin-export-1.6.0 prometheus-client-0.20.0 prompt-toolkit-3.0.43 psutil-5.9.8 ptyprocess-0.7.0 pure-eval-0.2.2 pyasn1-0.5.1 pycparser-2.21 pygments-2.17.2 pynacl-1.5.0 pyproject-hooks-1.0.0 pyrsistent-0.20.0 pytest-8.0.1 pytest-cov-4.1.0 pytest-instafail-0.5.0 pytest-xdist-3.5.0 python-dateutil-2.8.2 python-dotenv-0.21.1 python-json-logger-2.0.7 pytz-2024.1 pyyaml-5.3.1 pyzmq-24.0.1 qtconsole-5.5.1 qtpy-2.4.1 rapidfuzz-3.6.1 requests-2.31.0 requests-toolbelt-1.0.0 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 rsa-4.7.2 s3fs-0.4.2 s3transfer-0.10.0 send2trash-1.8.2 shellingham-1.5.4 six-1.16.0 sniffio-1.3.0 soupsieve-2.5 stack-data-0.6.3 terminado-0.18.0 texttable-1.7.0 tinycss2-1.2.1 tomlkit-0.12.3 tornado-6.4 tqdm-4.66.2 traitlets-5.14.1 trove-classifiers-2024.1.31 tzdata-2024.1 urllib3-2.0.7 virtualenv-20.25.1 wcwidth-0.2.13 webencodings-0.5.1 websocket-client-0.59.0 widgetsnbextension-4.0.10 xattr-0.10.1 zipp-3.17.0
# aws=aws-cli/1.32.47 Python/3.11.5 Darwin/23.3.0 botocore/1.34.47
# invoke=Invoke 2.2.0
# poetry=Poetry (version 1.7.1)
# docker-compose=docker-compose version 1.29.2, build unknown
dev_scripts/client_setup/build.sh:60: command not found: docker

[Process completed]

Seems my docker is not installed (docker-compose was installed successfully) Q: Is it because I used an M2 chip? (I don't want to use the docker desktop, though I'm pretty sure it can solve my problem)

~~Useful link: https://github.com/sorrentum/sorrentum/issues/607~~

samarth9008 commented 7 months ago

The useful link is unrelated. You need to install docker in your laptop to move forward. I think you can install docker without docker-desktop.

jayhsu0627 commented 7 months ago

@samarth9008 I've installed the env of source dev_scripts/client_setup/build.sh by ~~manually installed docker~~ Docker Desktop

brew install docker
brew link docker
brew install colima

However, a new command not found: complete issue pop-out by running source dev_scripts/setenv_amp.sh

# Configure env
which gh=/opt/homebrew/bin/gh
AM_AWS_PROFILE=am
AM_AWS_S3_BUCKET=alphamatic-data
AM_ECR_BASE_PATH=665840871993.dkr.ecr.us-east-1.amazonaws.com
CK_AWS_PROFILE=ck
CK_AWS_S3_BUCKET=cryptokaizen-data
TERM_PROGRAM_VERSION=453
i=invoke
ih='invoke --help'
il='invoke --list'
it='invoke traceback'
itpb='pbpaste | traceback_to_cfile.py -i - -o cfile'
run-help=man
which-command=whence
# Enable invoke autocompletion.
/Users/sheng***/src/sorrentum1/dev_scripts/invoke_completion.sh:33: command not found: complete

EDIT: MacOS M2 chip, Sonoma 14.3 After Google, I guess it's because I'm using a pure MacOS shell instead of zsh. So I try to install zsh

it works now

gwt='git worktree'
gwta='git worktree add'
gwtls='git worktree list'
gwtmv='git worktree move'
gwtrm='git worktree remove'
history=omz_history
i=invoke
ih='invoke --help'
il='invoke --list'
it='invoke traceback'
itpb='pbpaste | traceback_to_cfile.py -i - -o cfile'
l='ls -lah'
la='ls -lAh'
ll='ls -lh'
ls='ls -G'
lsa='ls -lah'
md='mkdir -p'
rd=rmdir
run-help=man
which-command=whence
# Enable invoke autocompletion.
==> SUCCESS <==

gpsaggese commented 7 months ago

Good job @jayhsu0627 investigating.

1) I use bash on my M2 mac and things are fine with complete

saggese@gpmac.local ==> RC: 1 <== venv:(amp.client_venv) branch:'master' ~/src/sorrentum1
> echo $SHELL
/bin/bash

saggese@gpmac.local venv:(amp.client_venv) branch:'master' ~/src/sorrentum1
> source dev_scripts/invoke_completion.sh
# Enable invoke autocompletion.

2) @samarth9008 can we update the doc clarifying that one needs Docker desktop of the GUI-less one

brew install docker
brew link docker
brew install colima

jayhsu0627 commented 7 months ago

I try to verify my docker installation by running i docker_jupyter from old_quick_start


INFO: > cmd='/Users/USERNAME/src/venv/amp.client_venv/bin/invoke docker_jupyter'
## docker_jupyter: 
22:05:55 - INFO  lib_tasks_docker.py docker_jupyter:1449                Assigned port is 5011
22:05:55 - WARN  <string> _raise_invalid_host:90                        Don't recognize host: host_os_name=Darwin, am_host_os_name=None
22:05:55 - INFO  lib_tasks_docker.py _docker_cmd:1306                   Pulling the latest version of Docker
## docker_pull: 
## docker_login: 
## _docker_login_ecr: 
  ... 
  ... The config profile (ck) could not be found
22:05:55 - INFO  lib_tasks_docker.py _docker_pull:227                   image='623860924167.dkr.ecr.eu-north-1.amazonaws.com/cmamp:dev'
docker pull 623860924167.dkr.ecr.eu-north-1.amazonaws.com/cmamp:dev
Error response from daemon: Head "https://623860924167.dkr.ecr.eu-north-1.amazonaws.com/v2/cmamp/manifests/dev": no basic auth credentials

(amp.client_venv) ➜  sorrentum1 git:(master) ✗ i docker_pull_dev_tools

INFO: > cmd='/Users/USERNAME/src/venv/amp.client_venv/bin/invoke docker_pull_dev_tools'
## docker_pull_dev_tools: 
## docker_login: 
## _docker_login_ecr: 
  ... 
  ... The config profile (ck) could not be found
22:08:09 - INFO  lib_tasks_docker.py _docker_pull:227                   image='623860924167.dkr.ecr.eu-north-1.amazonaws.com/dev_tools:prod'
docker pull 623860924167.dkr.ecr.eu-north-1.amazonaws.com/dev_tools:prod
Error response from daemon: Head "https://623860924167.dkr.ecr.eu-north-1.amazonaws.com/v2/dev_tools/manifests/prod": no basic auth credentials

But I got a bad AWS credentials error, I know I didn't setup an AWS id. Maybe it's the same as this warning?

WARNING: The AM_AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.

samarth9008 commented 7 months ago

Are you able to do i docker_pull and if thats successfull, i docker_bash?

samarth9008 commented 7 months ago

@gpsaggese

As far as I understand, when someone works from sorrentum, the image has a base path of dockerhub resitry that is, sorrentum/cmamp:dev and not the ECR registry path 623860924167.dkr.ecr.eu-north-1.amazonaws.com/cmamp:dev.

These paths are set in tasks.py file over here. Currently it is pointing to ECR base path but earlier when we do i docker_pull it would fetch the image from sorrentum/cmamp:dev instead 623860924167.dkr.ecr.eu-north-1.amazonaws.com/cmamp:dev.

Are you aware of any workaround, that we applied to make the image fetch from dockerhub instead of ECR for people working on sorrentum or dev_tools? This changes might have appeared after the integration.

jayhsu0627 commented 7 months ago

Are you able to do i docker_pull and if thats successfull, i docker_bash?

this is my i docker_pull

(amp.client_venv) ➜  sorrentum1 git:(master) ✗ i docker_pull
INFO: > cmd='/Users/username/src/venv/amp.client_venv/bin/invoke docker_pull'
## docker_pull: 
## docker_login: 
## _docker_login_ecr: 
  ... 
  ... The config profile (ck) could not be found
10:55:15 - INFO  lib_tasks_docker.py _docker_pull:227                   image='623860924167.dkr.ecr.eu-north-1.amazonaws.com/cmamp:dev'
docker pull 623860924167.dkr.ecr.eu-north-1.amazonaws.com/cmamp:dev
Error response from daemon: Head "https://623860924167.dkr.ecr.eu-north-1.amazonaws.com/v2/cmamp/manifests/dev": no basic auth credentials

and i docker_bash

(amp.client_venv) ➜  sorrentum1 git:(master) ✗  i docker_bash
INFO: > cmd='/Users/username/src/venv/amp.client_venv/bin/invoke docker_bash'
## docker_bash: 
10:56:13 - WARN  <string> _raise_invalid_host:90                        Don't recognize host: host_os_name=Darwin, am_host_os_name=None
10:56:13 - INFO  lib_tasks_docker.py _docker_cmd:1306                   Pulling the latest version of Docker
## docker_pull: 
## docker_login: 
## _docker_login_ecr: 
  ... 
  ... The config profile (ck) could not be found
10:56:13 - INFO  lib_tasks_docker.py _docker_pull:227                   image='623860924167.dkr.ecr.eu-north-1.amazonaws.com/cmamp:dev'
docker pull 623860924167.dkr.ecr.eu-north-1.amazonaws.com/cmamp:dev
Error response from daemon: Head "https://623860924167.dkr.ecr.eu-north-1.amazonaws.com/v2/cmamp/manifests/dev": no basic auth credentials

samarth9008 commented 7 months ago

You are not supposed to do docker login as you do not have access to our aws infra

However if you want to move forward for, locally change this line over here https://github.com/sorrentum/sorrentum/blob/d39076822ab9d6bf20f6b7e59c196a6b6589ecc3/tasks.py#L167

"CK_ECR_BASE_PATH": ECR_BASE_PATH -> "CK_ECR_BASE_PATH": "sorrentum"

samarth9008 commented 7 months ago

Pulling the latest master will resolve your changes. No need to make any changes anywhere

samarth9008 commented 7 months ago

The link to Read Quick start for developing is changed. Pls refer the updated link.

jayhsu0627 commented 7 months ago

You are not supposed to do docker login as you do not have access to our aws infra

However if you want to move forward for, locally change this line over here

https://github.com/sorrentum/sorrentum/blob/d39076822ab9d6bf20f6b7e59c196a6b6589ecc3/tasks.py#L167

"CK_ECR_BASE_PATH": ECR_BASE_PATH -> "CK_ECR_BASE_PATH": "sorrentum"

Thank you, samarth! I edited according to your suggest, but I got this error while running i docker_bash and i docker_jupyter

WARNING: AWS credential check failed: can't find /home/.aws/credentials file.
WARNING: AWS credential check failed: can't find /home/.aws/config file.
# Check AWS authentication setup
rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
 devops/docker_run/entrypoint.sh: line 109:    21 Trace/breakpoint trap   aws configure --profile am list
AM_CONTAINER_VERSION='1.12.0'
which python: /venv/bin/python
python -V: Python 3.9.5
helpers: <module 'helpers' from '/app/helpers/__init__.py'>
PATH=/app/dev_scripts/testing:/app/dev_scripts/notebooks:/app/dev_scripts/install:/app/dev_scripts/infra:/app/dev_scripts/git:/app/dev_scripts/documentation:/app/dev_scripts/aws:/app/dev_scripts:/app:.:/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH=/app:
entrypoint.sh: 'devops/docker_run/run_jupyter_server.sh'
Traceback (most recent call last):
  File "/venv/bin/invoke", line 8, in <module>
    sys.exit(program.run())
  File "/venv/lib/python3.9/site-packages/invoke/program.py", line 387, in run
    self.parse_collection()
  File "/venv/lib/python3.9/site-packages/invoke/program.py", line 479, in parse_collection
    self.load_collection()
  File "/venv/lib/python3.9/site-packages/invoke/program.py", line 716, in load_collection
    module, parent = loader.load(coll_name)
  File "/venv/lib/python3.9/site-packages/invoke/loader.py", line 91, in load
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/app/tasks.py", line 106, in <module>
    from oms.lib_tasks_binance import (  # isort: skip # noqa: F401  # pylint: disable=unused-import
  File "/app/oms/__init__.py", line 7, in <module>
    from oms.broker.broker import *  # pylint: disable=unused-import # NOQA
  File "/app/oms/broker/broker.py", line 22, in <module>
    import market_data as mdata
  File "/app/market_data/__init__.py", line 8, in <module>
    from market_data.im_client_market_data import *  # pylint: disable=unused-import # NOQA
  File "/app/market_data/im_client_market_data.py", line 15, in <module>
    import im_v2.common.data.client as icdc
  File "/app/im_v2/common/data/client/__init__.py", line 13, in <module>
    from im_v2.common.data.client.im_raw_data_client import *  # pylint: disable=unused-import # NOQA
  File "/app/im_v2/common/data/client/im_raw_data_client.py", line 23, in <module>
    import im_v2.common.db.db_utils as imvcddbut
  File "/app/im_v2/common/db/db_utils.py", line 24, in <module>
    import helpers.hsql_test as hsqltest
  File "/app/helpers/hsql_test.py", line 34, in <module>
    not henv.execute_repo_config_code("has_dind_support()")
  File "/app/helpers/henv.py", line 536, in execute_repo_config_code
    ret = eval(code_to_execute)
  File "<string>", line 1, in <module>
  File "<string>", line 208, in has_dind_support
AssertionError: Expected privileged mode: has_dind=False
is_cmamp_prod=False
is_dev4=False
is_dev_ck=False
is_ig_prod=False
is_inside_ci=False
is_mac=True
ERROR: 1

Possible reason https://github.com/sorrentum/sorrentum/issues/370?

gpsaggese commented 7 months ago

1) I see rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2 that makes me think that for some reason you might be using the x86 image + emulator and not the Apple Silicon one (which of course it's slower since it goes through the translation layer).

@samarth9008 we should have a check / warning about this. I'll file a bug.

gpsaggese commented 7 months ago

2) The error is because you don't have docker-in-docker enabled. This is an old way of doing things that we don't really use anymore.

In repo_config.py, you can set check_repo = False

198     # dind is supported on both Mac and GH Actions.
199     check_repo = os.environ.get("AM_REPO_CONFIG_CHECK", "True") != "False"
200     if check_repo:
201         if hserver.is_cmamp_prod():
202             assert not has_dind, "Not expected privileged mode"

I'll file another bug to make this check optional.

samarth9008 commented 7 months ago

Updated link for setup - https://github.com/sorrentum/sorrentum/blob/master/docs/onboarding/sorrentum.set_up_development_environment.how_to_guide.md

jayhsu0627 commented 7 months ago

The error is because you don't have docker-in-docker enabled. This is an old way of doing things that we don't really use anymore.

In repo_config.py, you can set check_repo = False
198     # dind is supported on both Mac and GH Actions.
199     check_repo = os.environ.get("AM_REPO_CONFIG_CHECK", "True") != "False"
200     if check_repo:
201         if hserver.is_cmamp_prod():
202             assert not has_dind, "Not expected privileged mode"
I'll file another bug to make this check optional.

After ignoring the dind check by setting check_repo = False, I can successfully get i docker_bash and i docker_bash running.

...
# Env vars:
  AM_AWS_ACCESS_KEY_ID=undef
  AM_AWS_DEFAULT_REGION=undef
  AM_AWS_PROFILE='am'
  AM_AWS_S3_BUCKET='alphamatic-data'
  AM_AWS_SECRET_ACCESS_KEY=undef
  AM_ECR_BASE_PATH='665840871993.dkr.ecr.us-east-1.amazonaws.com'
  AM_ENABLE_DIND='0'
  AM_FORCE_TEST_FAIL=''
  AM_HOST_NAME='sjxu'
  AM_HOST_OS_NAME='Darwin'
  AM_HOST_USER_NAME='username'
  AM_HOST_VERSION='23.3.0'
  AM_REPO_CONFIG_CHECK='True'
  AM_REPO_CONFIG_PATH=''
  AM_TELEGRAM_TOKEN=empty
  CI=''
  CK_AWS_ACCESS_KEY_ID=empty
  CK_AWS_DEFAULT_REGION=''
  CK_AWS_S3_BUCKET='cryptokaizen-data'
  CK_AWS_SECRET_ACCESS_KEY=empty
  CK_ECR_BASE_PATH='sorrentum'
  GH_ACTION_ACCESS_TOKEN=empty

22:06:26 - INFO  hcache.py clear_global_cache:294                       Before clear_global_cache: 'global mem' cache: path='/mnt/tmpfs/tmp.cache.mem', size=36.0 KB
22:06:26 - WARN  hcache.py clear_global_cache:295                       Resetting 'global mem' cache '/mnt/tmpfs/tmp.cache.mem'
22:06:26 - WARN  hcache.py clear_global_cache:305                       Destroying '/mnt/tmpfs/tmp.cache.mem' ...
22:06:26 - INFO  hcache.py clear_global_cache:321                       After clear_global_cache: 'global mem' cache: path='/mnt/tmpfs/tmp.cache.mem', size=nan
user_501@3bab0cb655bd:/app$

(amp.client_venv) ➜  sorrentum1 git:(master) ✗ i docker_jupyter 5080
INFO: > cmd='/Users/username/src/venv/amp.client_venv/bin/invoke docker_jupyter 5080'
No idea what '5080' is!
(amp.client_venv) ➜  sorrentum1 git:(master) ✗ i docker_jupyter -p 5080
INFO: > cmd='/Users/username/src/venv/amp.client_venv/bin/invoke docker_jupyter -p 5080'
## docker_jupyter: 
22:10:48 - INFO  lib_tasks_docker.py docker_jupyter:1449                Assigned port is 5080
22:10:48 - WARN  <string> _raise_invalid_host:90                        Don't recognize host: host_os_name=Darwin, am_host_os_name=None
22:10:48 - INFO  lib_tasks_docker.py _docker_cmd:1306                   Pulling the latest version of Docker
## docker_pull: 
## docker_login: 
## _docker_login_ecr: 
  ... 
  ... The config profile (ck) could not be found
22:10:49 - INFO  lib_tasks_docker.py _docker_pull:227                   image='sorrentum/cmamp:dev'
docker pull sorrentum/cmamp:dev
dev: Pulling from sorrentum/cmamp
Digest: sha256:bf774bd028a92dbff5b0efafc505a7e40c88419c70447b3f815f9ee53ac72473
Status: Image is up to date for sorrentum/cmamp:dev
docker.io/sorrentum/cmamp:dev
IMAGE=sorrentum/cmamp:dev \
PORT=5080 \
        docker-compose \
        --file /Users/username/src/sorrentum1/devops/compose/docker-compose.yml \
        --env-file devops/env/default.env \
        run \
        --rm \
        --name username.cmamp.jupyter_server.sorrentum1.20240226_221048 \
        --user $(id -u):$(id -g) \
        --service-ports \
        jupyter_server 
WARNING: The AM_AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AM_AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
WARNING: The AM_AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AM_FORCE_TEST_FAIL variable is not set. Defaulting to a blank string.
WARNING: The AM_TELEGRAM_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The CK_AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The CK_AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
WARNING: The CK_AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The CK_TELEGRAM_TOKEN variable is not set. Defaulting to a blank string.
Creating compose_jupyter_server_run ... done
##> devops/docker_run/entrypoint.sh
UID=501
GID=20
# Activate environment
##> devops/docker_run/setenv.sh
# Set PATH
PATH=/app/dev_scripts/testing:/app/dev_scripts/notebooks:/app/dev_scripts/install:/app/dev_scripts/infra:/app/dev_scripts/git:/app/dev_scripts/documentation:/app/dev_scripts/aws:/app/dev_scripts:/app:.:/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Set PYTHONPATH
PYTHONPATH=/app:
# Configure env
git --version: git version 2.25.1
/app
WARNING: AWS credential check failed: can't find /home/.aws/credentials file.
WARNING: AWS credential check failed: can't find /home/.aws/config file.
# Check AWS authentication setup
rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
 devops/docker_run/entrypoint.sh: line 109:    21 Trace/breakpoint trap   aws configure --profile am list
AM_CONTAINER_VERSION='1.12.0'
which python: /venv/bin/python
python -V: Python 3.9.5
helpers: <module 'helpers' from '/app/helpers/__init__.py'>
PATH=/app/dev_scripts/testing:/app/dev_scripts/notebooks:/app/dev_scripts/install:/app/dev_scripts/infra:/app/dev_scripts/git:/app/dev_scripts/documentation:/app/dev_scripts/aws:/app/dev_scripts:/app:.:/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH=/app:
entrypoint.sh: 'devops/docker_run/run_jupyter_server.sh'
WARNING: Skip checking since AM_REPO_CONFIG_CHECK='True'
INFO: > cmd='/venv/bin/invoke print_env'
22:11:01 - WARN  <string> _raise_invalid_host:90                        Don't recognize host: host_os_name=Linux, am_host_os_name=Darwin
WARNING: Skip checking since AM_REPO_CONFIG_CHECK='True'
(CVXPY) Feb 27 03:11:01 AM: Encountered unexpected exception importing solver GLPK:
ImportError("cannot import name 'glpk' from 'cvxopt' (/venv/lib/python3.9/site-packages/cvxopt/__init__.py)")
(CVXPY) Feb 27 03:11:01 AM: Encountered unexpected exception importing solver GLPK_MI:
ImportError("cannot import name 'glpk' from 'cvxopt' (/venv/lib/python3.9/site-packages/cvxopt/__init__.py)")
# Repo config:
  # repo_config.config
    enable_privileged_mode='False'
    get_docker_base_image_name='cmamp'
    get_docker_shared_group=''
    get_docker_user=''
    get_host_name='github.com'
    get_html_dir_to_url_mapping='{'s3://cryptokaizen-html': 'http://172.30.2.44'}'
    get_invalid_words='[]'
    get_name='//sorr'
    get_repo_map='{'sorr': 'sorrentum/sorrentum'}'
    get_shared_data_dirs='None'
    has_dind_support='False'
    has_docker_sudo='True'
    is_CK_S3_available='True'
    run_docker_as_root='False'
    skip_submodules_test='False'
    use_docker_db_container_name_to_connect='False'
    use_docker_network_mode_host='False'
    use_docker_sibling_containers='False'
    # hserver.config
      is_AM_S3_available()='True'
      is_dev4()='False'
      is_dev_ck()='False'
      is_inside_ci()='False'
      is_inside_docker()='True'
      is_mac(version='Catalina')='False'
      is_mac(version='Monterey')='False'
      is_mac(version='Ventura')='False'
# System signature:
  # Git
    branch_name='master'
    hash='95fb77aba'
    # Last commits:
      * 95fb77aba Samarth KaPatel update (#675)                                                     (    3 days ago) Fri Feb 23 16:42:39 2024  (HEAD -> master, origin/master, origin/HEAD)
      * d39076822 Vlad     fix from cmamp (#674)                                             (    4 days ago) Fri Feb 23 14:53:28 2024           
      * c3e351cbd GP Saggese AmpTask1786_Integrate_20240219_5 (#669)                           (    5 days ago) Wed Feb 21 22:27:11 2024           
  # Machine info
    system=Linux
    node name=a6171d0581e3
    release=6.6.12-linuxkit
    version=#1 SMP Thu Feb  8 06:36:34 UTC 2024
    machine=aarch64
    processor=aarch64
    cpu count=8
    cpu freq=None
    memory=svmem(total=8222334976, available=7053840384, percent=14.2, used=958861312, free=1380122624, active=1268436992, inactive=5147893760, buffers=79269888, cached=5804081152, shared=1527808, slab=303693824)
    disk usage=sdiskusage(total=62671097856, used=14573363200, free=44881018880, percent=24.5)
  # Packages
    python: 3.9.5
    cvxopt: 1.3.2
    cvxpy: 1.4.2
    gluonnlp: ?
    gluonts: ?
    joblib: 1.3.2
    mxnet: ?
    numpy: 1.26.0
    pandas: 2.1.1
    pyarrow: 14.0.2
    scipy: 1.11.3
    seaborn: 0.13.0
    sklearn: 1.3.1
    statsmodels: 0.14.0
# Env vars:
  AM_AWS_ACCESS_KEY_ID=undef
  AM_AWS_DEFAULT_REGION=undef
  AM_AWS_PROFILE='am'
  AM_AWS_S3_BUCKET='alphamatic-data'
  AM_AWS_SECRET_ACCESS_KEY=undef
  AM_ECR_BASE_PATH='665840871993.dkr.ecr.us-east-1.amazonaws.com'
  AM_ENABLE_DIND='0'
  AM_FORCE_TEST_FAIL=''
  AM_HOST_NAME='sjxu'
  AM_HOST_OS_NAME='Darwin'
  AM_HOST_USER_NAME='username'
  AM_HOST_VERSION='23.3.0'
  AM_REPO_CONFIG_CHECK='True'
  AM_REPO_CONFIG_PATH=''
  AM_TELEGRAM_TOKEN=empty
  CI=''
  CK_AWS_ACCESS_KEY_ID=empty
  CK_AWS_DEFAULT_REGION=''
  CK_AWS_S3_BUCKET='cryptokaizen-data'
  CK_AWS_SECRET_ACCESS_KEY=empty
  CK_ECR_BASE_PATH='sorrentum'
  GH_ACTION_ACCESS_TOKEN=empty

22:11:01 - INFO  hcache.py clear_global_cache:294                       Before clear_global_cache: 'global mem' cache: path='/mnt/tmpfs/tmp.cache.mem', size=36.0 KB
22:11:01 - WARN  hcache.py clear_global_cache:295                       Resetting 'global mem' cache '/mnt/tmpfs/tmp.cache.mem'
22:11:01 - WARN  hcache.py clear_global_cache:305                       Destroying '/mnt/tmpfs/tmp.cache.mem' ...
22:11:01 - INFO  hcache.py clear_global_cache:321                       After clear_global_cache: 'global mem' cache: path='/mnt/tmpfs/tmp.cache.mem', size=nan
##> devops/docker_run/run_jupyter_server.sh
> cmd=jupyter notebook --ip=* --port=5080 --allow-root --NotebookApp.token=''
[W 03:11:02.071 NotebookApp] Config option `default_jupytext_formats` not recognized by `LargeFileManager`.
[W 03:11:02.071 NotebookApp] Config option `preferred_jupytext_formats_save` not recognized by `LargeFileManager`.
[W 03:11:02.071 NotebookApp] Config option `outdated_text_notebook_margin` not recognized by `LargeFileManager`.  Did you mean `untitled_notebook`?
[I 03:11:02.089 NotebookApp] Writing notebook server cookie secret to /home/.local/share/jupyter/runtime/notebook_cookie_secret
[I 03:11:02.090 NotebookApp] Authentication of /metrics is OFF, since other authentication is disabled.

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[W 03:11:02.434 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[W 03:11:02.434 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using authentication. This is highly insecure and not recommended.
[I 03:11:02.437 NotebookApp] [jupyter_nbextensions_configurator] enabled 0.6.3
[I 03:11:02.486 NotebookApp] [Jupytext Server Extension] Deriving a JupytextContentsManager from LargeFileManager
[I 03:11:02.489 NotebookApp] Serving notebooks from local directory: /app
[I 03:11:02.489 NotebookApp] Jupyter Notebook 6.5.6 is running at:
[I 03:11:02.489 NotebookApp] http://a6171d0581e3:5080/
[I 03:11:02.489 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 03:11:02.492 NotebookApp] No web browser found: could not locate runnable browser.

However, I encountered another bug maybe: after close the jupyter notebook server by ctrl+c, the following port already allocated error pop-out. So I manually change the port to a new one 5080 by i docker_jupyter -p 5080.

(amp.client_venv) ➜  sorrentum1 git:(master) ✗ i docker_jupyter                    
INFO: > cmd='/Users/username/src/venv/amp.client_venv/bin/invoke docker_jupyter'
## docker_jupyter: 
22:10:07 - INFO  lib_tasks_docker.py docker_jupyter:1449                Assigned port is 5011
22:10:07 - WARN  <string> _raise_invalid_host:90                        Don't recognize host: host_os_name=Darwin, am_host_os_name=None
22:10:07 - INFO  lib_tasks_docker.py _docker_cmd:1306                   Pulling the latest version of Docker
## docker_pull: 
## docker_login: 
## _docker_login_ecr: 
  ... 
  ... The config profile (ck) could not be found
22:10:07 - INFO  lib_tasks_docker.py _docker_pull:227                   image='sorrentum/cmamp:dev'
docker pull sorrentum/cmamp:dev
dev: Pulling from sorrentum/cmamp
Digest: sha256:bf774bd028a92dbff5b0efafc505a7e40c88419c70447b3f815f9ee53ac72473
Status: Image is up to date for sorrentum/cmamp:dev
docker.io/sorrentum/cmamp:dev
IMAGE=sorrentum/cmamp:dev \
PORT=5011 \
        docker-compose \
        --file /Users/username/src/sorrentum1/devops/compose/docker-compose.yml \
        --env-file devops/env/default.env \
        run \
        --rm \
        --name username.cmamp.jupyter_server.sorrentum1.20240226_221007 \
        --user $(id -u):$(id -g) \
        --service-ports \
        jupyter_server 
WARNING: The AM_AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AM_AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
WARNING: The AM_AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AM_FORCE_TEST_FAIL variable is not set. Defaulting to a blank string.
WARNING: The AM_TELEGRAM_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The CK_AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The CK_AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
WARNING: The CK_AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The CK_TELEGRAM_TOKEN variable is not set. Defaulting to a blank string.
Creating compose_jupyter_server_run ... done
Error response from daemon: driver failed programming external connectivity on endpoint username.cmamp.jupyter_server.sorrentum1.20240226_221007 (586d2523365f8cc2a4f596a88d03e00a8066a016163ec6a8b4d12b11608310bb): Bind for 0.0.0.0:5011 failed: port is already allocated

i run_fast_tests partial result

======================================================= 21 failed, 2222 passed, 204 skipped, 470 deselected, 2 xfailed in 87.85s (0:01:27) =======================================================
22:39:17 @ 2024-02-26 05:38:50 - INFO  hcache.py clear_global_cache:294 Before clear_global_cache: 'global mem' cache: path='/mnt/tmpfs/tmp.cache.mem', size=40.0 KB
22:39:17 @ 2024-02-26 05:38:50 - WARN  hcache.py clear_global_cache:295 Resetting 'global mem' cache '/mnt/tmpfs/tmp.cache.mem'
22:39:17 @ 2024-02-26 05:38:50 - WARN  hcache.py clear_global_cache:305 Destroying '/mnt/tmpfs/tmp.cache.mem' ...
22:39:17 @ 2024-02-26 05:38:50 - INFO  hcache.py clear_global_cache:321 After clear_global_cache: 'global mem' cache: path='/mnt/tmpfs/tmp.cache.mem', size=nan
ERROR: 1

@gpsaggese @sonaalKant Is there any way I can build an M2 docker? If it's not necessary, I think I'm ready to be assigned a warm-up issue.

sonaalKant commented 7 months ago

I have assigned you the warm up issue https://github.com/sorrentum/sorrentum/issues/695

gpsaggese commented 7 months ago

1) The docker image is multi-arch so it should pull automatically the right one. In fact I don't see the warning from Rosetta about running x86 images

2) If you close the container, it should release the port. Otherwise you can force kill the container

kaizen-ai / kaizenflow

On-board Shengjie Xu #665