search

exasol / ai-lab

Development environment for data science developers
MIT License
3 stars 0 forks source link

AWS Code Build fails with error message #303

Closed ckunki closed 1 month ago

ckunki commented 1 month ago

AWS CodeBuild Log

In setup of test_jupyter_with_ec2_based_on_new_ami 

    @pytest.fixture(scope="session")
    def new_ec2_from_ami():

Line 97

change_password(host=ec2_instance_description.public_dns_name, user='ubuntu',
                                curr_pass=default_password, new_password=new_password)

Error message

/root/.cache/pypoetry/virtualenvs/exasol-ai-lab-X0J5PNhP-py3.10/lib/python3.10/site-packages/ paramiko/auth_handler.py:263: BadAuthenticationType

ckunki commented 1 month ago

Currently, the AI-Lab code uses lock_passwd from module users-and-groups, potentially, module set-passwords could also play a role.

AI Lab already contains Ansible task Enable SSH password authentication in file reset_password_tasks.yml, but the current assumption is that later this is disabled by cloud init, again.

Observation: Inside the Docker Container the file /etc/ssh/sshd_config doesn't exist anymore.

ckunki commented 1 month ago

Proposal:

- name: Enable SSH password authentication
  become: "{{need_sudo}}"
  lineinfile:
    dest: /etc/cloud/cloud.cfg
    regexp: '^(\s*)ssh_pwauth: false'
    line: '\1ssh_pwauth: true'
    state: present
    backrefs: yes
ckunki commented 1 month ago

Proposal for investigation:

ckunki commented 1 month ago

It seems already Ansible installation reports an error:

/usr/bin/apt-get -y
  -o "Dpkg::Options::=--force-confdef"
  -o "Dpkg::Options::=--force-confold"
  install 'network-manager=1.36.4-2ubuntu1'
  -o APT::Install-Recommends=no'

failed: E: Unable to correct problems, you have held broken packages. stdout:

Reading package lists... Building dependency tree... Reading state information...

Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: network-manager : Depends: libnm0 (= 1.36.4-2ubuntu1) but 1.36.6-0ubuntu2 is to be installed

ckunki commented 1 month ago

I will update network-manager as described above

ckunki commented 1 month ago

I could retrieve files /etc/cloud/cloud.cfg and /etc/ssh/sshd_config now.

/etc/cloud/cloud.cfg contains 

system_info:
  default_user:
    name: ubuntu
    lock_passwd: True

/etc/ssh/sshd_config contains 

#PasswordAuthentication yes
ckunki commented 1 month ago

Calling poetry run python exasol/ds/sandbox/main.py reset-password now displays

Unable to encrypt nor hash, passlib must be installed. No module named 'passlib'. Unable to encrypt nor hash, passlib must be installed. No module named 'passlib'"

See https://stackoverflow.com/questions/78525098/ansible-unable-to-encrypt-nor-hash-passlib-must-be-installed

ckunki commented 1 month ago

Added dependency to passlib: poetry add passlib, unfortunatly caused some downgrades of other dependencies: 

  • Downgrading urllib3 (2.2.2 -> 2.2.1)
  • Downgrading botocore (1.34.137 -> 1.34.125)
  • Installing commonmark (0.9.1)
  • Downgrading fastjsonschema (2.20.0 -> 2.19.1)
  • Downgrading plux (1.11.0 -> 1.10.0)
  • Downgrading psutil (6.0.0 -> 5.9.8)
  • Downgrading pydantic-core (2.20.0 -> 2.18.4)
  • Downgrading rich (13.7.1 -> 12.6.0)
  • Downgrading s3transfer (0.10.2 -> 0.10.1)
  • Downgrading boto3 (1.34.137 -> 1.34.125)
  • Installing jsonpickle (3.2.1)
  • Installing pbr (6.0.0)
  • Downgrading pydantic (2.8.0 -> 2.7.4)
  • Downgrading setuptools (70.2.0 -> 70.0.0)
  • Downgrading ansible-core (2.17.1 -> 2.16.7)
  • Downgrading invoke (2.2.0 -> 1.7.3)
  • Installing jschema-to-python (1.2.3)
  • Installing junit-xml (1.9)
  • Downgrading networkx (3.3 -> 2.8.8)
  • Installing pathlib2 (2.3.7.post1)
  • Downgrading pytest (8.2.2 -> 7.4.4)
  • Installing sarif-om (1.0.4)
  • Downgrading ansible (10.1.0 -> 9.6.1)
  • Downgrading cfn-lint (1.4.2 -> 0.65.1)
  • Downgrading fabric (3.2.2 -> 2.7.1)
  • Downgrading importlib-metadata (7.2.1 -> 7.1.0)
  • Installing passlib (1.7.4)
  • Downgrading pytest-check-links (0.10.1 -> 0.9.3)
  • Downgrading tenacity (8.4.2 -> 8.3.0)
ckunki commented 1 month ago

I was now able to run poetry run python exasol/ds/sandbox/main.py reset-password without obvious errors.

After commenting out some parts of file reset_password_tasks.yml, I was still able to log in via ssh key file but not with password.

File contents: 

$ grep -H lock /etc/cloud/cloud.cfg ; grep -H Passwo /etc/ssh/sshd_config
/etc/cloud/cloud.cfg:    lock_passwd: False
/etc/ssh/sshd_config:PasswordAuthentication yes
ckunki commented 1 month ago

Thanks to @tkilias we identified to additionally need sshd option KbdInteractiveAuthentication yes. Additionally, we were able to remove dependency passlib and replace it by python standard module crypt. This enables to revert the downgrades of the other packages mentioned above.

ckunki commented 1 month ago

Python test based on fabric / paramiko still failed as with

paramiko.ssh_exception.BadAuthenticationType: Bad authentication type; allowed types: ['publickey', 'keyboard-interactive']

It could be possible to use keyboard-interactive authentication with fabric, see

On the other hand, I found the following at https://superuser.com/a/1828947, which turned out to be true in our case, too:

sneaky file /etc/ssh/sshd_config.d/60-cloudimg-settings.conf that had a single line PasswordAuthentication no

So I updated ansible task to modify all files in folder /etc/ssh/sshd_config.d, too, using file globbing for remote files, as described here: https://stackoverflow.com/questions/33543551/

ckunki commented 1 month ago

I manually verified login with KbdInteractiveAuthentication no in /etc/ssh/sshd_config to be successful.