Bug: failed to extract fingerprints

[benibr]

I'm running in a strange new error with current master branch when generateconf.py is called by Docker build

 => ERROR [migrid install_mig 8/9] RUN ./generateconfs.py --source=.     --destination=generated-confs     --base_fqdn=storage.deic.dk     --public_fqdn=www.storage.deic.dk     --mig_cert_fqdn=     --ext_cert_fqdn=     --mig_oid_fq  0.5s ------
 > [migrid install_mig 8/9] RUN ./generateconfs.py --source=.     --destination=generated-confs     --base_fqdn=storage.deic.dk     --public_fqdn=www.storage.deic.dk     --mig_cert_fqdn=     --ext_cert_fqdn=     --mig_oid_fqdn=ext.storage
.deic.dk     --ext_oid_fqdn=oid.storage.deic.dk     --ext_oidc_fqdn=     --sid_fqdn=sid.storage.deic.dk     --io_fqdn=     --user=mig --group=mig --log_level=info     --support_email="mig" --smtp_sender=""     --admin_email="XXXXXXX" --admin_list="XXXXXXXX"     --smtp_se
rver="localhost"     --apache_version=2.4     --apache_etc=/etc/httpd     --apache_run=/var/run/httpd     --apache_lock=/var/lock/subsys/httpd     --apache_log=/var/log/httpd     --openssh_version=7.4     --mig_code=/home/mig/mig     --mi
g_state=/home/mig/state     --mig_certs=/etc/httpd/MiG-certificates     --hg_path="/usr/bin/hg"      --hgweb_scripts="/usr/share/doc/mercurial"     --trac_admin_path=""     --trac_ini_path=""     --openid_address=ext.storage.deic.dk     -
-sftp_address=sftp.storage.deic.dk     --sftp_subsys_address=${SFTP_SUBSYS_DOMAIN}     --ftps_address=ftps.storage.deic.dk     --davs_address=webdavs.storage.deic.dk     --public_http_port=80 --public_https_port=443     --mig_oid_port=443
 --ext_oid_port=443     --ext_oidc_port=443 --mig_cert_port=443     --ext_cert_port=443 --sid_port=443     --sftp_port=2222 --sftp_subsys_port=22222     --sftp_show_port=2222     --davs_port=4443 --davs_show_port=4443     --ftps_ctrl_port
=8021 --ftps_ctrl_show_port=21     --ftps_pasv_ports=8100:8400     --openid_port=8443 --openid_show_port=443     --io_account_expire=False     --mig_oid_title="Non-KU/UCPH" --ext_oid_title="KU/UCPH"     --mig_oid_provider=https://ext.stor
age.deic.dk/openid/     --ext_oid_provider=https://openid.ku.dk/     --ext_oidc_provider_meta_url=unset     --ext_oidc_client_name="unset"     --ext_oidc_client_id="unset"     --ext_oidc_scope="profile email"     --ext_oidc_remote_user_cl
aim="sub"     --ext_oidc_pass_claim_as="unset"     --enable_openid=True --enable_wsgi=True     --enable_sftp=True --enable_sftp_subsys=True     --enable_davs=True --enable_ftps=True     --enable_sharelinks=True --enable_transfers=True
 --enable_duplicati=True --enable_seafile=False     --enable_sandboxes=False --enable_vmachines=False     --enable_crontab=True --enable_jobs=False     --enable_resources=False --enable_events=False     --enable_freeze=True --enable_imnot
ify=False     --enable_cracklib=True --enable_twofactor=True     --enable_twofactor_strict_address=False     --enable_peers=True --peers_mandatory=True     --peers_explicit_fields="full_name email"     --peers_contact_hint="employed at UC
PH and authorized to invite you as peer"     --enable_notify=True --enable_preview=False     --enable_workflows=False --enable_hsts=True     --enable_vhost_certs=True --enable_verify_certs=True     --enable_jupyter=False --enable_migadmin
=True     --enable_gdp=False --gdp_email_notify=False     --gdp_id_scramble=safe_hash --gdp_path_scramble=safe_encrypt     --password_policy=MEDIUM     --jupyter_services="dag.http://dag.test"     --jupyter_services_desc="{'dag': '/home/m
ig/state/wwwpublic/dag_desc.html'}"     --prefer_python3=True     --user_clause=User --group_clause=Group     --listen_clause='#Listen'     --serveralias_clause='ServerAlias' --alias_field=email     --dhparams_path=/etc/httpd/MiG-certific
ates/dhparams.pem     --daemon_keycert=/etc/httpd/MiG-certificates/combined.pem     --daemon_pubkey=/etc/httpd/MiG-certificates/combined.pub     --daemon_pubkey_from_dns=False     --daemon_show_address=     --signup_methods="migoid extoid
"     --login_methods="migoid extoid"     --distro=centos --user_interface="V3"     --skin="erda-ucph-science"     --title="DeiC Storage" --short_title="DSTOR"     --digest_salt="XXXXX" --crypto_salt="XXXXX"     --vgrid_label="Workgroup" --peers_permit="role:.*(vip|tap)"     --vgrid_creators="role:.*(vip|tap)" --vgrid_managers="distinguished_name:.*"     --default_vgrid_links="files web"     --advanced_vgrid_links="fil
es web scm tracker workflows monitor"     --auto_add_cert_user=True     --auto_add_oid_user=True     --auto_add_oidc_user=True     --cert_valid_days=365 --oid_valid_days=365     --generic_valid_days=365     --default_menu="home files vgri
ds archives settings setup logout" --user_menu="sharelinks crontab transfers people downloads peers docs migadmin"     --status_system_match=="ANY"     --secscan_addr=""     --imnotify_address= --imnotify_channel=     --imnotify_username=
 --imnotify_password=     --external_doc=https://sourceforge.net/p/migrid/wiki     --extra_userpage_scripts=""     --extra_userpage_styles=""     --sftp_max_sessions=32     --apache_worker_procs=256 --wsgi_procs=25:
0.382 System has not been booted with systemd as init system (PID 1). Can't operate.
0.382 Failed to create bus connection: Host is down
0.449 # Creating confs with:
0.449 source: .
0.449 destination: generated-confs
[...]
0.449 enable_twofactor_strict_address: False
0.449 enable_peers: True
0.449 peers_mandatory: True
0.449 enable_cracklib: True
0.449 enable_openid: True
0.449 enable_gravatars: DEFAULT
0.449 enable_sitestatus: DEFAULT
0.449 daemon_pubkey_from_dns: False
0.449 seafile_ro_access: DEFAULT
0.449 public_use_https: DEFAULT
0.449 prefer_python3: True
0.449 io_account_expire: False
0.449 gdp_email_notify: False
0.449 ERROR: failed to extract sha256 fingerprint of /etc/httpd/MiG-certificates/combined.pem: a bytes-like object is required, not 'str'
0.449 ERROR: failed to extract fingerprints of /etc/httpd/MiG-certificates/combined.pub : a bytes-like object is required, not 'str'
0.449 Error stripping __SEAFILE_CCNET_ID__: [b'39af9cfb3c4d0c4246be12ef4394283499707f20']
0.449 Traceback (most recent call last):
0.449   File "./generateconfs.py", line 349, in <module>
0.449     conf = generate_confs(**settings)
0.449   File "/home/mig/mig/shared/install.py", line 1803, in generate_confs
0.449     fill_template(in_path, out_path, user_dict, strip_trailing_space)
0.449   File "/home/mig/mig/shared/install.py", line 87, in fill_template
0.449     raise exc
0.449   File "/home/mig/mig/shared/install.py", line 84, in fill_template
0.449     contents = re.sub(variable + suffix, value, contents)
0.449   File "/usr/lib64/python3.6/re.py", line 191, in sub
0.449     return _compile(pattern, flags).sub(repl, string, count)
0.449 TypeError: sequence item 1: expected str instance, bytes found

The certs exists and havn't been changed since last deployment. It works with experimental branch so it might be a Python related error.

[jonasbardino]

Yes, I've seen similar when forcing python3 on edge. Please note that python3 strictly requires the git experimental branch to build/run. It probably started showing in relation to the recent changes in Dockerfile regarding automatic selection of preferred python and git branch to support building each of the different centos and rocky flavors without adjusting the provided development.env and production.env files accordingly. A valid MIG_GIT_BRANCH and PREFER_PYTHON3 combo should now automatically get selected when left unset in your .env like in the sample envs. So please leave them both alone or explicitly set PREFER_PYTHON3=False if you set MIG_GIT_BRANCH=edge. Does that help or can you elaborate on the env settings?

[jonasbardino]

I suppose this issue really belongs at docker-migrid, but if we're looking at the plain migrid setup the cause is correspondingly the use of prefer_python3=True with a branch other than experimental.

[benibr]

Ah okay that makes sense and would explain why the behaviour recently changed. I sometime deployed master for some testing but then I make sure that I also use always experimental for this.

Anyway, might be good to have this error message archived here, but I guess we can close this issue.