nilsteampassnet / TeamPass

Collaborative Passwords Manager
https://www.teampass.net
1.63k stars 532 forks source link

Endless loop while upgrading from 3.0.10 to 3.1.1 #4156

Closed xavier8854 closed 2 months ago

xavier8854 commented 2 months ago

Steps to reproduce

  1. Launch upgrade
  2. Wait for an hour
  3. Browser crashes (memory ?)

Expected behaviour

Smooth upgrade

Actual behaviour

Stuck in "Performing operation populateitems Table_CreatedAt Loop # 8950" and counting

Server configuration

Operating system: RHEL 9 Web server: Apache 2.4.57 Database: MariaDB 10.11.7 PHP version: 8.1.27 Teampass version: 3.0.10 -> 3.1.1 Teampass configuration file:

<?php
global $SETTINGS;
$SETTINGS = array (
    'max_latest_items' => '10',
    'enable_favourites' => '1',
    'show_last_items' => '1',
    'enable_pf_feature' => '0',
    'log_connections' => '1',
    'log_accessed' => '1',
    'time_format' => 'H:i:s',
    'date_format' => 'd/m/Y',
    'duplicate_folder' => '0',
    'item_duplicate_in_same_folder' => '0',
    'duplicate_item' => '0',
    'number_of_used_pw' => '3',
    'manager_edit' => '1',
    'cpassman_dir' => '/var/www/html/teampass',
    'cpassman_url' => 'https://REDACTED/teampass',
    'favicon' => 'https://REDACTED/teampass/favicon.ico',
    'path_to_upload_folder' => '/var/www/html/teampass/upload',
    'path_to_files_folder' => '/var/www/html/teampass/files',
    'url_to_files_folder' => 'https://REDACTED/teampass/files',
    'activate_expiration' => '0',
    'pw_life_duration' => '0',
    'maintenance_mode' => '0',
    'enable_sts' => '0',
    'encryptClientServer' => '1',
    'teampass_version' => '3.0.10',
    'ldap_mode' => '0',
    'ldap_type' => 'OpenLDAP',
    'ldap_suffix' => '0',
    'ldap_domain_dn' => '0',
    'ldap_domain_controler' => '0',
    'ldap_user_attribute' => 'cn',
    'ldap_ssl' => '0',
    'ldap_tls' => '0',
    'ldap_search_base' => '0',
    'ldap_port' => '636',
    'richtext' => '0',
    'allow_print' => '1',
    'roles_allowed_to_print' => '0',
    'show_description' => '1',
    'anyone_can_modify' => '0',
    'anyone_can_modify_bydefault' => '0',
    'nb_bad_authentication' => '0',
    'utf8_enabled' => '1',
    'restricted_to' => '0',
    'restricted_to_roles' => '0',
    'enable_send_email_on_user_login' => '0',
    'enable_user_can_create_folders' => '0',
    'insert_manual_entry_item_history' => '0',
    'enable_kb' => '0',
    'enable_email_notification_on_item_shown' => '0',
    'enable_email_notification_on_user_pw_change' => '0',
    'custom_logo' => '',
    'custom_login_text' => '',
    'default_language' => 'french',
    'send_stats' => '0',
    'send_statistics_items' => 'stat_country;stat_users;stat_items;stat_items_shared;stat_folders;stat_folders_shared;stat_admins;stat_managers;stat_ro;stat_mysqlversion;stat_phpversion;stat_teampassversion;stat_languages;stat_kb;stat_suggestion;stat_customfields;stat_api;stat_2fa;stat_agses;stat_duo;stat_ldap;stat_syslog;stat_stricthttps;stat_fav;stat_pf;',
    'send_stats_time' => '1711285550',
    'get_tp_info' => '1',
    'send_mail_on_user_login' => '0',
    'sending_emails' => '0',
    'nb_items_by_query' => 'auto',
    'enable_delete_after_consultation' => '0',
    'enable_personal_saltkey_cookie' => '0',
    'personal_saltkey_cookie_duration' => '31',
    'email_smtp_server' => 'localhost',
    'email_smtp_auth' => '',
    'email_auth_username' => '',
    'email_auth_pwd' => '',
    'email_port' => '25',
    'email_security' => '',
    'email_server_url' => '',
    'email_from' => 'teampass@REDACTED',
    'email_from_name' => 'Teampass Administrator',
    'pwd_maximum_length' => '40',
    'google_authentication' => '0',
    'delay_item_edition' => '0',
    'allow_import' => '1',
    'proxy_ip' => '',
    'proxy_port' => '',
    'upload_maxfilesize' => '10mb',
    'upload_docext' => 'doc,docx,dotx,xls,xlsx,xltx,rtf,csv,txt,pdf,ppt,pptx,pot,dotx,xltx',
    'upload_imagesext' => 'jpg,jpeg,gif,png',
    'upload_pkgext' => '7z,rar,tar,zip',
    'upload_otherext' => 'sql,xml',
    'upload_imageresize_options' => '1',
    'upload_imageresize_width' => '800',
    'upload_imageresize_height' => '600',
    'upload_imageresize_quality' => '90',
    'use_md5_password_as_salt' => '0',
    'ga_website_name' => 'TeamPass for ChangeMe',
    'api' => '0',
    'subfolder_rights_as_parent' => '0',
    'show_only_accessible_folders' => '0',
    'enable_suggestion' => '0',
    'otv_expiration_period' => '7',
    'default_session_expiration_time' => '60',
    'duo' => '0',
    'enable_server_password_change' => '0',
    'ldap_object_class' => '0',
    'bck_script_path' => '/var/www/html/teampass/backups',
    'bck_script_filename' => 'bck_teampass',
    'syslog_enable' => '0',
    'syslog_host' => 'localhost',
    'syslog_port' => '514',
    'manager_move_item' => '0',
    'create_item_without_password' => '0',
    'otv_is_enabled' => '0',
    'agses_authentication_enabled' => '0',
    'item_extra_fields' => '0',
    'saltkey_ante_2127' => 'none',
    'migration_to_2127' => 'done',
    'files_with_defuse' => 'done',
    'timezone' => 'UTC',
    'enable_attachment_encryption' => '1',
    'personal_saltkey_security_level' => '50',
    'ldap_new_user_is_administrated_by' => '0',
    'disable_show_forgot_pwd_link' => '0',
    'offline_key_level' => '0',
    'enable_http_request_login' => '0',
    'ldap_and_local_authentication' => '0',
    'secure_display_image' => '1',
    'upload_zero_byte_file' => '0',
    'upload_all_extensions_file' => '0',
    'bck_script_passkey' => 'EDITED',
    'admin_2fa_required' => '1',
    'password_overview_delay' => '4',
    'copy_to_clipboard_small_icons' => '1',
    'duo_ikey' => '',
    'duo_skey' => '',
    'duo_host' => '',
    'duo_failmode' => 'secure',
    'roles_allowed_to_print_select' => '',
    'clipboard_life_duration' => '30',
    'mfa_for_roles' => '',
    'tree_counters' => '0',
    'settings_offline_mode' => '1',
    'settings_tree_counters' => '0',
    'enable_massive_move_delete' => '0',
    'email_debug_level' => '2',
    'ga_reset_by_user' => '',
    'onthefly-backup-key' => '',
    'onthefly-restore-key' => '',
    'ldap_user_dn_attribute' => 'REDACTED',
    'ldap_dn_additional_user_dn' => '',
    'ldap_user_object_filter' => '',
    'ldap_bdn' => 'REDACTED',
    'ldap_hosts' => 'REDACTED',
    'ldap_password' => 'REDACTED',
    'ldap_username' => 'REDACTED',
    'api_token_duration' => '60',
    'last_folder_change' => '',
    'enable_tasks_manager' => '1',
    'task_maximum_run_time' => '300',
    'tasks_manager_refreshing_period' => '20',
    'maximum_number_of_items_to_treat' => '100',
    'ldap_tls_certifacte_check' => 'LDAP_OPT_X_TLS_NEVER',
    'enable_tasks_log' => '1',
    'upgrade_timestamp' => '1713877550',
    'enable_ad_users_with_ad_groups' => '0',
    'enable_ad_user_auto_creation' => '0',
    'ldap_group_object_filter' => 'uid',
    'ldap_guid_attibute' => 'objectguid',
    'sending_emails_job_frequency' => '2',
    'user_keys_job_frequency' => '1',
    'items_statistics_job_frequency' => '5',
    'users_personal_folder_task' => '',
    'clean_orphan_objects_task' => '',
    'purge_temporary_files_task' => '',
    'rebuild_config_file' => '',
    'reload_cache_table_task' => '',
    'maximum_session_expiration_time' => '60',
    'items_ops_job_frequency' => '1',

Updated from an older Teampass or fresh install: Update from 3.0.10 to 3.1.1

Client configuration

Browser: Fierfox or CHrome Operating system: Ubuntu 23.10

Logs

Web server error log

[24/Apr/2024:15:01:45 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:45 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:46 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:46 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:47 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:47 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:48 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:49 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:49 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:50 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:50 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:51 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:51 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:52 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:52 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:53 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:53 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:54 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:54 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:55 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:55 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:56 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:57 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:57 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:58 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:58 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:59 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:01:59 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:00 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:00 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:01 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:02 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:02 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:03 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:03 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:04 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:04 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:05 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:05 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:06 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:06 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:07 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:08 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:08 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:09 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:09 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:10 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:10 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:11 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
[24/Apr/2024:15:02:11 +0200] 172.29.187.220 TLSv1.3 TLS_AES_256_GCM_SHA384 "POST /teampass/install/upgrade_operations.php HTTP/1.1" 52
^C

Screenshot from 2024-04-24 14-19-28

xavier8854 commented 2 months ago

Ended with a clean reinstall

ivancasanovaicp commented 1 month ago

Hello Xavier, I have the same problem. How did you reinstall keeping all configuration and DB?

Thanks

xavier8854 commented 1 month ago

Hi, Unfortunately, I didn't, reinstalled from scratch 3.0.10, and reimported CSV

ivancasanovaicp commented 4 weeks ago

How did you export all the passwords?

xavier8854 commented 4 weeks ago

In version 2.X, there were afaik, no export function, so I copy/pasted them from Teampass 2.X to a CSV.

ivancasanovaicp commented 4 weeks ago

I just solved the problem.

searching in Teampass/install/upgrade_operations.php.

function populateItemsTable_CreatedAt($pre, $post_nb) { global $db_link; // loop on items - created_at $items = mysqli_query( $db_link, "select i.id as id, ls.date as datetime from " . $pre . "items as i inner join " . $pre . "log_items as ls on ls.id_item = i.id WHERE ls.action = 'at_creation' AND i.created_at IS NULL LIMIT " . $post_nb.";" ); while ($item = mysqli_fetch_assoc($items)) { if (empty((string) $item['datetime']) === false && is_null($item['datetime']) === false) { // update created_at field mysqli_query( $db_link, "UPDATE " . $pre . "items SET created_at = '".$item['datetime']."' WHERE id = ".$item['id'] ); } }

// Is it finished?
$remainingItems = mysqli_num_rows(
    mysqli_query(
        $db_link,
        "SELECT * FROM `" . $pre . "items` WHERE created_at IS NULL"
    )
);
return $remainingItems > 0 ? 0 : 1;

This function looks for elements that do not have a creation date in the teampass_items table to paste the data from the log_items table.

After modifying them, checks that there are no objects in the "items" table that have "created_at" as Null.

But no query checks for "ls.date" in "log_items".

Manually updating all the dates in my database on the "items" table, and launching that last query "SELECT * FROM " . $pre . "items WHERE created_at IS NULL". I have seen that I still found 2 items created which did not have a date, but at the same time, the first script that joins the values ​​of both tables, did NOT find these items.

I don't know why but I had two passwords created with no creation date set.

The problem is that the script does not consider that this could happen, since in the first query it does not look for NULL values ​​in ls.date in the "log_items" table.

I hope to help @nilsteampassnet in future versions or someone who needs it.