[BUG] rpm-ostree core dumps on 36

hkaancaliskan commented 2 years ago

Describe the bug I'm using pretty much basic, unmodified silverblue. I've rebased to 36 and rpm-ostree crashes when i run rpm-ostree upgrade. I've used this article to rebase to 36 https://fedoramagazine.org/how-to-rebase-to-fedora-linux-35-on-silverblue/

To Reproduce Idk how to reproduce, i've just rebased to 36 and rpm-ostree never worked on 36. On 35, everything works perfectly.

Expected behavior rpm-ostree works fine

OS version:

[kaan@fedora ~]$ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  fedora:fedora/35/x86_64/silverblue
                   Version: 35.20220222.0 (2022-02-22T03:30:21Z)
                BaseCommit: 83726b81a2e5d5a2464dbdcf2b34d74bf6ca022dab281ca361adbcdfa6d2efec
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
                      Diff: 6 upgraded, 1172 downgraded, 31 removed, 93 added
       RemovedBasePackages: firefox 97.0-1.fc35 gnome-tour 41~rc-1.fc35
                    Pinned: yes

? fedora:fedora/36/x86_64/silverblue
                   Version: 36.20220221.n.0 (2022-02-21T08:06:40Z)
                BaseCommit: 604d1d4b9e5f2ba2ec308d907b30d836ab131c2e689500701ebe939526f91547
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
       RemovedBasePackages: firefox 96.0-1.fc36 gnome-tour 41~rc-2.fc36

Additional context

[kaan@fedora ~]$ rpm-ostree upgrade 
note: automatic updates (stage) are enabled
error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`.

[kaan@fedora ~]$ systemctl status rpm-ostreed.service 
? rpm-ostreed.service - rpm-ostree System Management Daemon
     Loaded: loaded (/usr/lib/systemd/system/rpm-ostreed.service; static)
     Active: inactive (dead)
       Docs: man:rpm-ostree(1)

?ub 22 20:28:24 fedora rpm-ostree[1803]: Process [pid: 6098 uid: 1000 unit: user@1000.service] connected to transaction progress
?ub 22 20:28:25 fedora systemd[1]: rpm-ostreed.service: Main process exited, code=dumped, status=11/SEGV
?ub 22 20:28:25 fedora systemd[1]: rpm-ostreed.service: Failed with result 'core-dump'.
?ub 22 20:28:25 fedora systemd[1]: rpm-ostreed.service: Consumed 9.539s CPU time.
?ub 22 20:28:42 fedora systemd[1]: Starting rpm-ostreed.service - rpm-ostree System Management Daemon...
?ub 22 20:28:42 fedora rpm-ostree[7133]: Reading config file '/etc/rpm-ostreed.conf'
?ub 22 20:28:43 fedora rpm-ostree[7133]: In idle state; will auto-exit in 61 seconds
?ub 22 20:28:43 fedora systemd[1]: Started rpm-ostreed.service - rpm-ostree System Management Daemon.
?ub 22 20:29:44 fedora systemd[1]: rpm-ostreed.service: Deactivated successfully.
?ub 22 20:29:44 fedora systemd[1]: rpm-ostreed.service: Consumed 2.405s CPU time.

    [kaan@fedora ~]$ journalctl -xb -u rpm-ostreed.service 
    ?ub 22 20:27:38 fedora systemd[1]: Starting rpm-ostreed.service - rpm-ostree System Management Daemon...
    ?? Subject: A start job for unit rpm-ostreed.service has begun execution
    ?? Defined-By: systemd
    ?? Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ?? 
    ?? A start job for unit rpm-ostreed.service has begun execution.
    ?? 
    ?? The job identifier is 2294.
    ?ub 22 20:27:38 fedora rpm-ostree[1803]: Reading config file '/etc/rpm-ostreed.conf'
    ?ub 22 20:27:39 fedora rpm-ostree[1803]: In idle state; will auto-exit in 64 seconds
    ?ub 22 20:27:39 fedora systemd[1]: Started rpm-ostreed.service - rpm-ostree System Management Daemon.
    ?? Subject: A start job for unit rpm-ostreed.service has finished successfully
    ?? Defined-By: systemd
    ?? Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ?? 
    ?? A start job for unit rpm-ostreed.service has finished successfully.
    ?? 
    ?? The job identifier is 2294.
    ?ub 22 20:27:39 fedora rpm-ostree[1803]: Allowing active client :1.70 (uid 1000)
    ?ub 22 20:27:39 fedora rpm-ostree[1803]: client(id:gnome-software dbus:1.70 unit:app-gnome-gnome\x2dsoftware\x2dservice-1513.scope uid:1000) added; new total=1
    ?ub 22 20:27:51 fedora rpm-ostree[1803]: Allowing active client :1.86 (uid 1000)
    ?ub 22 20:27:51 fedora rpm-ostree[1803]: client(id:cli dbus:1.86 unit:vte-spawn-d4994bdf-3e75-4992-a1d5-c17996517ac1.scope uid:1000) added; new total=2
    ?ub 22 20:27:52 fedora rpm-ostree[1803]: client(id:cli dbus:1.86 unit:vte-spawn-d4994bdf-3e75-4992-a1d5-c17996517ac1.scope uid:1000) vanished; remaining=1
    ?ub 22 20:28:23 fedora rpm-ostree[1803]: Allowing active client :1.89 (uid 1000)
    ?ub 22 20:28:23 fedora rpm-ostree[1803]: client(id:cli dbus:1.89 unit:vte-spawn-b94efc4e-6f5e-41cc-842a-7c6f9530a63b.scope uid:1000) added; new total=2
    ?ub 22 20:28:24 fedora rpm-ostree[1803]: Locked sysroot
    ?ub 22 20:28:24 fedora rpm-ostree[1803]: Initiated txn Upgrade for client(id:cli dbus:1.89 unit:vte-spawn-b94efc4e-6f5e-41cc-842a-7c6f9530a63b.scope uid:1000): /org/pr>
    ?ub 22 20:28:24 fedora rpm-ostree[1803]: Process [pid: 6098 uid: 1000 unit: user@1000.service] connected to transaction progress
    ?ub 22 20:28:25 fedora systemd[1]: rpm-ostreed.service: Main process exited, code=dumped, status=11/SEGV
    ?? Subject: Unit process exited
    ?? Defined-By: systemd
    ?? Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ?? 
    ?? An ExecStart= process belonging to unit rpm-ostreed.service has exited.
    ?? 
    ?? The process' exit code is 'dumped' and its exit status is 11.
    ?ub 22 20:28:25 fedora systemd[1]: rpm-ostreed.service: Failed with result 'core-dump'.
    ?? Subject: Unit failed
    ?? Defined-By: systemd
    ?? Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ?? 
    ?? The unit rpm-ostreed.service has entered the 'failed' state with result 'core-dump'.
    ?ub 22 20:28:25 fedora systemd[1]: rpm-ostreed.service: Consumed 9.539s CPU time.
    ?? Subject: Resources consumed by unit runtime
    ?? Defined-By: systemd
    ?? Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ?? 
    ?? The unit rpm-ostreed.service completed and consumed the indicated resources.
    ?ub 22 20:28:42 fedora systemd[1]: Starting rpm-ostreed.service - rpm-ostree System Management Daemon...
    ?? Subject: A start job for unit rpm-ostreed.service has begun execution
    ?? Defined-By: systemd
    ?? Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ?? 
    ?? A start job for unit rpm-ostreed.service has begun execution.
    ?? 
    ?? The job identifier is 2551.
    ?ub 22 20:28:42 fedora rpm-ostree[7133]: Reading config file '/etc/rpm-ostreed.conf'
    ?ub 22 20:28:43 fedora rpm-ostree[7133]: In idle state; will auto-exit in 61 seconds
    ?ub 22 20:28:43 fedora systemd[1]: Started rpm-ostreed.service - rpm-ostree System Management Daemon.
    ?? Subject: A start job for unit rpm-ostreed.service has finished successfully
    ?? Defined-By: systemd
    ?? Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ?? 
    ?? A start job for unit rpm-ostreed.service has finished successfully.
    ?? 
    ?? The job identifier is 2551.
    ?ub 22 20:29:44 fedora systemd[1]: rpm-ostreed.service: Deactivated successfully.
    ?? Subject: Unit succeeded
    ?? Defined-By: systemd
    ?? Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ?? 
    ?? The unit rpm-ostreed.service has successfully entered the 'dead' state.
    ?ub 22 20:29:44 fedora systemd[1]: rpm-ostreed.service: Consumed 2.405s CPU time.
    ?? Subject: Resources consumed by unit runtime
    ?? Defined-By: systemd
    ?? Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ?? 
    ?? The unit rpm-ostreed.service completed and consumed the indicated resources.

[kaan@fedora ~]$ sudo coredumpctl info
           PID: 1802 (rpm-ostree)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2022-02-22 21:28:28 +03 (46s ago)
  Command Line: /usr/bin/rpm-ostree start-daemon
    Executable: /usr/bin/rpm-ostree
 Control Group: /system.slice/rpm-ostreed.service
          Unit: rpm-ostreed.service
         Slice: system.slice
       Boot ID: 278ef914cfce43b59e54d016af5c38a5
    Machine ID: 67d340ffbfe74a2cb742e1138a063f33
      Hostname: fedora
       Storage: /var/lib/systemd/coredump/core.rpm-ostree.0.278ef914cfce43b59e54d016af5c38a5.1802.1645554508000000.zst (present)
     Disk Size: 798.1K
       Message: Process 1802 (rpm-ostree) of user 0 dumped core.

                Module linux-vdso.so.1 with build-id 42b5a520b08fcdd8bb1d434e51a29e0fbad7134c
                Module libnss_resolve.so.2 with build-id 74cb82260f0e39b720154cc57852ee07d3aa612b
                Metadata for module libnss_resolve.so.2 owned by FDO found: {
                        "type" : "rpm",
                        "name" : "systemd",
                        "version" : "250.3-4.fc36",
                        "architecture" : "x86_64",
                        "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                }

                Module libnss_mdns4_minimal.so.2 with build-id d2f6978d873a3f16c4a4da32c136ebe995550eff
                Metadata for module libnss_mdns4_minimal.so.2 owned by FDO found: {
                        "type" : "rpm",
                        "name" : "nss-mdns",
                        "version" : "0.15.1-4.fc36",
                        "architecture" : "x86_64",
                        "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                }

                Module libnss_myhostname.so.2 with build-id 347b028b15782fddc705758aa30a16dd07e43195
                Metadata for module libnss_myhostname.so.2 owned by FDO found: {
                        "type" : "rpm",
                        "name" : "systemd",
                        "version" : "250.3-4.fc36",
                        "architecture" : "x86_64",
                        "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                }

                Module libcrypt.so.2 with build-id 9924d3a82ca8fa6b94bc574a6520bcf84ccf831f
                Stack trace of thread 2570:
                #0  0x00007f678908f184 EVP_CIPHER_get_nid (libcrypto.so.3 + 0x185184)
                #1  0x00007f6789099630 EVP_CIPHER_get_type (libcrypto.so.3 + 0x18f630)
                #2  0x00007f67890c1f4f get_legacy_cipher_names (libcrypto.so.3 + 0x1b7f4f)
                #3  0x00007f67890b0637 OPENSSL_LH_doall_arg (libcrypto.so.3 + 0x1a6637)
                #4  0x00007f67890d082b OBJ_NAME_do_all (libcrypto.so.3 + 0x1c682b)
                #5  0x00007f67890c1a9a ossl_namemap_stored (libcrypto.so.3 + 0x1b7a9a)
                #6  0x00007f6789197691 inner_evp_generic_fetch.constprop.0 (libcrypto.so.3 + 0x28d691)
                #7  0x00007f6789099f64 EVP_RAND_fetch (libcrypto.so.3 + 0x18ff64)
                #8  0x00007f67890ec8d3 RAND_get0_primary (libcrypto.so.3 + 0x1e28d3)
                #9  0x00007f67890ecae7 RAND_status (libcrypto.so.3 + 0x1e2ae7)
                #10 0x00007f6789a5724e ossl_connect_step1 (libcurl.so.4 + 0x6d24e)
                #11 0x00007f6789a5ed87 ossl_connect_common (libcurl.so.4 + 0x74d87)
                #12 0x00007f6789a60988 Curl_ssl_connect_nonblocking (libcurl.so.4 + 0x76988)
                #13 0x00007f6789a16dc6 Curl_http_connect (libcurl.so.4 + 0x2cdc6)
                #14 0x00007f6789a33e0f multi_runsingle (libcurl.so.4 + 0x49e0f)
                #15 0x00007f6789a37159 multi_socket (libcurl.so.4 + 0x4d159)
                #16 0x00007f6789a372ff curl_multi_socket_action (libcurl.so.4 + 0x4d2ff)
                #17 0x00007f6788c79366 event_cb (libostree-1.so.1 + 0x9a366)
                #18 0x00007f67896aa1bf g_main_context_dispatch (libglib-2.0.so.0 + 0x551bf)
                #19 0x00007f67896ff2d8 g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xaa2d8)
                #20 0x00007f67896a7b40 g_main_context_iteration (libglib-2.0.so.0 + 0x52b40)
                #21 0x00007f6788c7a75d _ostree_fetcher_mirrored_request_to_membuf (libostree-1.so.1 + 0x9b75d)
                #22 0x00007f6788c31426 fetch_mirrored_uri_contents_utf8_sync.lto_priv.0 (libostree-1.so.1 + 0x52426)
                #23 0x00007f6788c31536 fetch_uri_contents_utf8_sync (libostree-1.so.1 + 0x52536)
                #24 0x00007f6788c3953a compute_effective_mirrorlist.lto_priv.0 (libostree-1.so.1 + 0x5a53a)
                #25 0x00007f6788c3ccdd ostree_repo_pull_with_options (libostree-1.so.1 + 0x5dcdd)
                #26 0x000056527732eccb rpmostree_sysroot_upgrader_pull_base (rpm-ostree + 0x788ccb)
                #27 0x00005652772f6072 _ZL26deploy_transaction_executeP22_RpmostreedTransactionP13_GCancellablePP7_GError (rpm-ostree + 0x750072)
                #28 0x00005652772ef80a _ZL26transaction_execute_threadP6_GTaskPvS1_P13_GCancellable (rpm-ostree + 0x74980a)
                #29 0x00007f678989e8e3 g_task_thread_pool_thread (libgio-2.0.so.0 + 0xb18e3)
                #30 0x00007f67896d6d22 g_thread_pool_thread_proxy.lto_priv.0 (libglib-2.0.so.0 + 0x81d22)
                #31 0x00007f67896d4322 g_thread_proxy (libglib-2.0.so.0 + 0x7f322)
                #32 0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #33 0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1809:
                #0  0x00007f6788aca6bd syscall (libc.so.6 + 0x10f6bd)
                #1  0x0000565276e65469 _ZN11parking_lot7condvar7Condvar19wait_until_internal17h04e8ab238aafc360E (rpm-ostree + 0x2bf469)
                #2  0x0000565277287f55 _ZN66_$LT$tokio..runtime..park..Parker$u20$as$u20$tokio..park..Park$GT$4park17hc53a12c880e07f1fE (rpm-ostree + 0x6e1f55)
                #3  0x00005652772976e9 _ZN5tokio7runtime11thread_pool6worker7Context12park_timeout17hc280b11cb01774eeE (rpm-ostree + 0x6f16e9)
                #4  0x0000565277296eeb _ZN5tokio7runtime11thread_pool6worker7Context3run17h80c00d05fde720f4E (rpm-ostree + 0x6f0eeb)
                #5  0x0000565277295f26 _ZN5tokio7runtime11thread_pool6worker3run17hba88ed1ab50dbbddE (rpm-ostree + 0x6eff26)
                #6  0x000056527728de60 _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hbc4af7fed5066a9bE (rpm-ostree + 0x6e7e60)
                #7  0x000056527727c8a0 _ZN5tokio7runtime8blocking4pool5Inner3run17h8f14cb156513fd2aE (rpm-ostree + 0x6d68a0)
                #8  0x0000565277281d82 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h959a1e4479cbb28bE.llvm.8865693629275206576 (rpm-ostree + 0x6dbd82)
                #9  0x000056527729a1bd _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h9b004e6e56af8776E (rpm-ostree + 0x6f41bd)
                #10 0x000056527725bf25 _ZN3std3sys4unix6thread6Thread3new12thread_start17he54490bbe3ccfc36E (rpm-ostree + 0x6b5f25)
                #11 0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #12 0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1802:
                #0  0x00007f6788aca6bd syscall (libc.so.6 + 0x10f6bd)
                #1  0x0000565276e65469 _ZN11parking_lot7condvar7Condvar19wait_until_internal17h04e8ab238aafc360E (rpm-ostree + 0x2bf469)
                #2  0x0000565277293d55 _ZN75_$LT$tokio..park..thread..CachedParkThread$u20$as$u20$tokio..park..Park$GT$4park17h10d8ff43e0b87819E (rpm-ostree + 0x6edd55)
                #3  0x0000565276f8d4cc _ZN5tokio4park6thread16CachedParkThread8block_on17h19085c1c099daec2E (rpm-ostree + 0x3e74cc)
                #4  0x0000565276fbbf84 _ZN5tokio7runtime11thread_pool10ThreadPool8block_on17h122cbf14fcb341a8E (rpm-ostree + 0x415f84)
                #5  0x0000565276fc8853 _ZN5tokio7runtime7Runtime8block_on17h0c6d2b2291c84a70E (rpm-ostree + 0x422853)
                #6  0x000056527700c2ea _ZN10rpm_ostree4main17h9226972d5a7b1290E (rpm-ostree + 0x4662ea)
                #7  0x0000565276fe0f73 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h5afd6e7fa3954bfbE (rpm-ostree + 0x43af73)
                #8  0x00005652770287e9 _ZN3std2rt10lang_start28_$u7b$$u7b$closure$u7d$$u7d$17h1103f8371612cea3E.llvm.17699875111697412146 (rpm-ostree + 0x4827e9)
                #9  0x0000565277253159 _ZN3std2rt19lang_start_internal17h9a77cb2d023ab68bE (rpm-ostree + 0x6ad159)
                #10 0x000056527700c6c8 main (rpm-ostree + 0x4666c8)
                #11 0x00007f67889e8590 __libc_start_call_main (libc.so.6 + 0x2d590)
                #12 0x00007f67889e8649 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2d649)
                #13 0x0000565276ceca95 _start (rpm-ostree + 0x146a95)

                Stack trace of thread 1810:
                #0  0x00007f6788aca6bd syscall (libc.so.6 + 0x10f6bd)
                #1  0x0000565276e65469 _ZN11parking_lot7condvar7Condvar19wait_until_internal17h04e8ab238aafc360E (rpm-ostree + 0x2bf469)
                #2  0x0000565277287f55 _ZN66_$LT$tokio..runtime..park..Parker$u20$as$u20$tokio..park..Park$GT$4park17hc53a12c880e07f1fE (rpm-ostree + 0x6e1f55)
                #3  0x00005652772976e9 _ZN5tokio7runtime11thread_pool6worker7Context12park_timeout17hc280b11cb01774eeE (rpm-ostree + 0x6f16e9)
                #4  0x0000565277296eeb _ZN5tokio7runtime11thread_pool6worker7Context3run17h80c00d05fde720f4E (rpm-ostree + 0x6f0eeb)
                #5  0x0000565277295f26 _ZN5tokio7runtime11thread_pool6worker3run17hba88ed1ab50dbbddE (rpm-ostree + 0x6eff26)
                #6  0x000056527728de60 _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hbc4af7fed5066a9bE (rpm-ostree + 0x6e7e60)
                #7  0x000056527727c8a0 _ZN5tokio7runtime8blocking4pool5Inner3run17h8f14cb156513fd2aE (rpm-ostree + 0x6d68a0)
                #8  0x0000565277281d82 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h959a1e4479cbb28bE.llvm.8865693629275206576 (rpm-ostree + 0x6dbd82)
                #9  0x000056527729a1bd _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h9b004e6e56af8776E (rpm-ostree + 0x6f41bd)
                #10 0x000056527725bf25 _ZN3std3sys4unix6thread6Thread3new12thread_start17he54490bbe3ccfc36E (rpm-ostree + 0x6b5f25)
                #11 0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #12 0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1807:
                #0  0x00007f6788aca6bd syscall (libc.so.6 + 0x10f6bd)
                #1  0x0000565276e65469 _ZN11parking_lot7condvar7Condvar19wait_until_internal17h04e8ab238aafc360E (rpm-ostree + 0x2bf469)
                #2  0x0000565277287f55 _ZN66_$LT$tokio..runtime..park..Parker$u20$as$u20$tokio..park..Park$GT$4park17hc53a12c880e07f1fE (rpm-ostree + 0x6e1f55)
                #3  0x00005652772976e9 _ZN5tokio7runtime11thread_pool6worker7Context12park_timeout17hc280b11cb01774eeE (rpm-ostree + 0x6f16e9)
                #4  0x0000565277296eeb _ZN5tokio7runtime11thread_pool6worker7Context3run17h80c00d05fde720f4E (rpm-ostree + 0x6f0eeb)
                #5  0x0000565277295f26 _ZN5tokio7runtime11thread_pool6worker3run17hba88ed1ab50dbbddE (rpm-ostree + 0x6eff26)
                #6  0x000056527728de60 _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hbc4af7fed5066a9bE (rpm-ostree + 0x6e7e60)
                #7  0x000056527727c8a0 _ZN5tokio7runtime8blocking4pool5Inner3run17h8f14cb156513fd2aE (rpm-ostree + 0x6d68a0)
                #8  0x0000565277281d82 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h959a1e4479cbb28bE.llvm.8865693629275206576 (rpm-ostree + 0x6dbd82)
                #9  0x000056527729a1bd _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h9b004e6e56af8776E (rpm-ostree + 0x6f41bd)
                #10 0x000056527725bf25 _ZN3std3sys4unix6thread6Thread3new12thread_start17he54490bbe3ccfc36E (rpm-ostree + 0x6b5f25)
                #11 0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #12 0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1812:
                #0  0x00007f6788aca6bd syscall (libc.so.6 + 0x10f6bd)
                #1  0x0000565276e65469 _ZN11parking_lot7condvar7Condvar19wait_until_internal17h04e8ab238aafc360E (rpm-ostree + 0x2bf469)
                #2  0x0000565277287f55 _ZN66_$LT$tokio..runtime..park..Parker$u20$as$u20$tokio..park..Park$GT$4park17hc53a12c880e07f1fE (rpm-ostree + 0x6e1f55)
                #3  0x00005652772976e9 _ZN5tokio7runtime11thread_pool6worker7Context12park_timeout17hc280b11cb01774eeE (rpm-ostree + 0x6f16e9)
                #4  0x0000565277296eeb _ZN5tokio7runtime11thread_pool6worker7Context3run17h80c00d05fde720f4E (rpm-ostree + 0x6f0eeb)
                #5  0x0000565277295f26 _ZN5tokio7runtime11thread_pool6worker3run17hba88ed1ab50dbbddE (rpm-ostree + 0x6eff26)
                #6  0x000056527728de60 _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hbc4af7fed5066a9bE (rpm-ostree + 0x6e7e60)
                #7  0x000056527727c8a0 _ZN5tokio7runtime8blocking4pool5Inner3run17h8f14cb156513fd2aE (rpm-ostree + 0x6d68a0)
                #8  0x0000565277281d82 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h959a1e4479cbb28bE.llvm.8865693629275206576 (rpm-ostree + 0x6dbd82)
                #9  0x000056527729a1bd _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h9b004e6e56af8776E (rpm-ostree + 0x6f41bd)
                #10 0x000056527725bf25 _ZN3std3sys4unix6thread6Thread3new12thread_start17he54490bbe3ccfc36E (rpm-ostree + 0x6b5f25)
                #11 0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #12 0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1811:
                #0  0x00007f6788aca6bd syscall (libc.so.6 + 0x10f6bd)
                #1  0x0000565276e65469 _ZN11parking_lot7condvar7Condvar19wait_until_internal17h04e8ab238aafc360E (rpm-ostree + 0x2bf469)
                #2  0x0000565277287f55 _ZN66_$LT$tokio..runtime..park..Parker$u20$as$u20$tokio..park..Park$GT$4park17hc53a12c880e07f1fE (rpm-ostree + 0x6e1f55)
                #3  0x00005652772976e9 _ZN5tokio7runtime11thread_pool6worker7Context12park_timeout17hc280b11cb01774eeE (rpm-ostree + 0x6f16e9)
                #4  0x0000565277296eeb _ZN5tokio7runtime11thread_pool6worker7Context3run17h80c00d05fde720f4E (rpm-ostree + 0x6f0eeb)
                #5  0x0000565277295f26 _ZN5tokio7runtime11thread_pool6worker3run17hba88ed1ab50dbbddE (rpm-ostree + 0x6eff26)
                #6  0x000056527728de60 _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hbc4af7fed5066a9bE (rpm-ostree + 0x6e7e60)
                #7  0x000056527727c8a0 _ZN5tokio7runtime8blocking4pool5Inner3run17h8f14cb156513fd2aE (rpm-ostree + 0x6d68a0)
                #8  0x0000565277281d82 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h959a1e4479cbb28bE.llvm.8865693629275206576 (rpm-ostree + 0x6dbd82)
                #9  0x000056527729a1bd _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h9b004e6e56af8776E (rpm-ostree + 0x6f41bd)
                #10 0x000056527725bf25 _ZN3std3sys4unix6thread6Thread3new12thread_start17he54490bbe3ccfc36E (rpm-ostree + 0x6b5f25)
                #11 0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #12 0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1808:
                #0  0x00007f6788ad0dde epoll_wait (libc.so.6 + 0x115dde)
                #1  0x0000565277283d85 _ZN5tokio2io6driver6Driver4turn17h93f5170f677566d6E.llvm.80244947646851454 (rpm-ostree + 0x6ddd85)
                #2  0x000056527728d2bc _ZN5tokio4time6driver15Driver$LT$P$GT$13park_internal17h47e9121493a5b62bE.llvm.6465144139353969602 (rpm-ostree + 0x6e72bc)
                #3  0x00005652772876f1 _ZN78_$LT$tokio..park..either..Either$LT$A$C$B$GT$$u20$as$u20$tokio..park..Park$GT$4park17hd49b541f369319d7E (rpm-ostree + 0x6e1>
                #4  0x0000565277287e94 _ZN66_$LT$tokio..runtime..park..Parker$u20$as$u20$tokio..park..Park$GT$4park17hc53a12c880e07f1fE (rpm-ostree + 0x6e1e94)
                #5  0x00005652772976e9 _ZN5tokio7runtime11thread_pool6worker7Context12park_timeout17hc280b11cb01774eeE (rpm-ostree + 0x6f16e9)
                #6  0x0000565277296eeb _ZN5tokio7runtime11thread_pool6worker7Context3run17h80c00d05fde720f4E (rpm-ostree + 0x6f0eeb)
                #7  0x0000565277295f26 _ZN5tokio7runtime11thread_pool6worker3run17hba88ed1ab50dbbddE (rpm-ostree + 0x6eff26)
                #8  0x000056527728de60 _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hbc4af7fed5066a9bE (rpm-ostree + 0x6e7e60)
                #9  0x000056527727c8a0 _ZN5tokio7runtime8blocking4pool5Inner3run17h8f14cb156513fd2aE (rpm-ostree + 0x6d68a0)
                #10 0x0000565277281d82 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h959a1e4479cbb28bE.llvm.8865693629275206576 (rpm-ostree + 0x6dbd82)
                #11 0x000056527729a1bd _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h9b004e6e56af8776E (rpm-ostree + 0x6f41bd)
                #12 0x000056527725bf25 _ZN3std3sys4unix6thread6Thread3new12thread_start17he54490bbe3ccfc36E (rpm-ostree + 0x6b5f25)
                #13 0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #14 0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1816:
                #0  0x00007f6788ac4faf __poll (libc.so.6 + 0x109faf)
                #1  0x00007f67896ff24d g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xaa24d)
                #2  0x00007f67896a7b40 g_main_context_iteration (libglib-2.0.so.0 + 0x52b40)
                #3  0x00007f67896a7b91 glib_worker_main (libglib-2.0.so.0 + 0x52b91)
                #4  0x00007f67896d4322 g_thread_proxy (libglib-2.0.so.0 + 0x7f322)
                #5  0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #6  0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1814:
                #0  0x00007f6788aca6bd syscall (libc.so.6 + 0x10f6bd)
                #1  0x0000565276e65469 _ZN11parking_lot7condvar7Condvar19wait_until_internal17h04e8ab238aafc360E (rpm-ostree + 0x2bf469)
                #2  0x0000565277287f55 _ZN66_$LT$tokio..runtime..park..Parker$u20$as$u20$tokio..park..Park$GT$4park17hc53a12c880e07f1fE (rpm-ostree + 0x6e1f55)
                #3  0x00005652772976e9 _ZN5tokio7runtime11thread_pool6worker7Context12park_timeout17hc280b11cb01774eeE (rpm-ostree + 0x6f16e9)
                #4  0x0000565277296eeb _ZN5tokio7runtime11thread_pool6worker7Context3run17h80c00d05fde720f4E (rpm-ostree + 0x6f0eeb)
                #5  0x0000565277295f26 _ZN5tokio7runtime11thread_pool6worker3run17hba88ed1ab50dbbddE (rpm-ostree + 0x6eff26)
                #6  0x000056527728de60 _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hbc4af7fed5066a9bE (rpm-ostree + 0x6e7e60)
                #7  0x000056527727c8a0 _ZN5tokio7runtime8blocking4pool5Inner3run17h8f14cb156513fd2aE (rpm-ostree + 0x6d68a0)
                #8  0x0000565277281d82 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h959a1e4479cbb28bE.llvm.8865693629275206576 (rpm-ostree + 0x6dbd82)
                #9  0x000056527729a1bd _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h9b004e6e56af8776E (rpm-ostree + 0x6f41bd)
                #10 0x000056527725bf25 _ZN3std3sys4unix6thread6Thread3new12thread_start17he54490bbe3ccfc36E (rpm-ostree + 0x6b5f25)
                #11 0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #12 0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1813:
                #0  0x00007f6788aca6bd syscall (libc.so.6 + 0x10f6bd)
                #1  0x0000565276e65469 _ZN11parking_lot7condvar7Condvar19wait_until_internal17h04e8ab238aafc360E (rpm-ostree + 0x2bf469)
                #2  0x0000565277287f55 _ZN66_$LT$tokio..runtime..park..Parker$u20$as$u20$tokio..park..Park$GT$4park17hc53a12c880e07f1fE (rpm-ostree + 0x6e1f55)
                #3  0x00005652772976e9 _ZN5tokio7runtime11thread_pool6worker7Context12park_timeout17hc280b11cb01774eeE (rpm-ostree + 0x6f16e9)
                #4  0x0000565277296eeb _ZN5tokio7runtime11thread_pool6worker7Context3run17h80c00d05fde720f4E (rpm-ostree + 0x6f0eeb)
                #5  0x0000565277295f26 _ZN5tokio7runtime11thread_pool6worker3run17hba88ed1ab50dbbddE (rpm-ostree + 0x6eff26)
                #6  0x000056527728de60 _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hbc4af7fed5066a9bE (rpm-ostree + 0x6e7e60)
                #7  0x000056527727c8a0 _ZN5tokio7runtime8blocking4pool5Inner3run17h8f14cb156513fd2aE (rpm-ostree + 0x6d68a0)
                #8  0x0000565277281d82 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h959a1e4479cbb28bE.llvm.8865693629275206576 (rpm-ostree + 0x6dbd82)
                #9  0x000056527729a1bd _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h9b004e6e56af8776E (rpm-ostree + 0x6f41bd)
                #10 0x000056527725bf25 _ZN3std3sys4unix6thread6Thread3new12thread_start17he54490bbe3ccfc36E (rpm-ostree + 0x6b5f25)
                #11 0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #12 0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 5475:
                #0  0x00007f6788aca6bd syscall (libc.so.6 + 0x10f6bd)
                #1  0x00007f67896faa90 g_cond_wait_until (libglib-2.0.so.0 + 0xa5a90)
                #2  0x00007f6789679f01 g_async_queue_pop_intern_unlocked (libglib-2.0.so.0 + 0x24f01)
                #3  0x00007f678967a095 g_async_queue_timeout_pop (libglib-2.0.so.0 + 0x25095)
                #4  0x00007f67896d6bd5 g_thread_pool_thread_proxy.lto_priv.0 (libglib-2.0.so.0 + 0x81bd5)
                #5  0x00007f67896d4322 g_thread_proxy (libglib-2.0.so.0 + 0x7f322)
                #6  0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #7  0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1817:
                #0  0x00007f6788ac4faf __poll (libc.so.6 + 0x109faf)
                #1  0x00007f67896ff24d g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xaa24d)
                #2  0x00007f67896a98ef g_main_loop_run (libglib-2.0.so.0 + 0x548ef)
                #3  0x00007f6789900f2a gdbus_shared_thread_func.lto_priv.0 (libgio-2.0.so.0 + 0x113f2a)
                #4  0x00007f67896d4322 g_thread_proxy (libglib-2.0.so.0 + 0x7f322)
                #5  0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #6  0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)

                Stack trace of thread 1815:
                #0  0x00007f6788ac4faf __poll (libc.so.6 + 0x109faf)
                #1  0x00007f67896ff24d g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xaa24d)
                #2  0x00007f67896a7b40 g_main_context_iteration (libglib-2.0.so.0 + 0x52b40)
                #3  0x00005652772ea14c rpmostree_builtin_start_daemon (rpm-ostree + 0x74414c)
                #4  0x00005652772ddd10 _ZN12rpmostreecxx14rpmostree_mainEN4rust10cxxbridge15SliceIKNS1_3StrEEE (rpm-ostree + 0x737d10)
                #5  0x00005652772d415b rpmostreecxx$cxxbridge1$rpmostree_main (rpm-ostree + 0x72e15b)
                #6  0x0000565276fa3194 _ZN102_$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$4poll17ha80ca7290c98>
                #7  0x00005652770308fd _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17h09dfed5466f47c5bE (rpm-ostree + 0x48a8fd)
                #8  0x000056527727c8a0 _ZN5tokio7runtime8blocking4pool5Inner3run17h8f14cb156513fd2aE (rpm-ostree + 0x6d68a0)
                #9  0x0000565277281d82 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h959a1e4479cbb28bE.llvm.8865693629275206576 (rpm-ostree + 0x6dbd82)
                #10 0x000056527729a1bd _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h9b004e6e56af8776E (rpm-ostree + 0x6f41bd)
                #11 0x000056527725bf25 _ZN3std3sys4unix6thread6Thread3new12thread_start17he54490bbe3ccfc36E (rpm-ostree + 0x6b5f25)
                #12 0x00007f6788a4c0b7 start_thread (libc.so.6 + 0x910b7)
                #13 0x00007f6788ad1800 __clone3 (libc.so.6 + 0x116800)
                ELF object binary architecture: AMD x86-64

jlebon commented 2 years ago

hkaancaliskan commented 2 years ago

ostree remote refs fedora returns While fetching https://ostree.fedoraproject.org/summary.sig: [35] SSL connect error.

juhp commented 2 years ago

I am assuming this is an upgrade issue from SB 35? I installed Silverblue 36 yesterday (36.20220225.n.0) in a VM and rpm-ostree seems to work fine there for me. And I just updated to 36.20220226.n.0 with no issues.

Anyway probably a good time to start testing upgrades as we are approaching Beta.

hkaancaliskan commented 2 years ago

@juhp yeah, only updated from latest stable S35 to S36 Commands i used to upgrade: 1- sudo ostree admin pin 0 2- rpm-ostree rebase fedora:fedora/36/x86_64/silverblue

~~Is there a possibility that it's somehow related to repos? I have 4 repos enabled rn. Fedora, fedora-testing, fedora-modular and modular-testing. Is there any other i need to enable?~~

Tried with 36 stable, testing and rawhide. All have the same problem.

juhp commented 2 years ago

Afraid I can't reproduce here on a clean and updated SB 35 install, after rebasing without any errors I got :

Deployments:
  fedora:fedora/36/x86_64/silverblue
                   Version: 36.20220227.n.0 (2022-02-27T08:03:47Z)
                    Commit: bc865956472db6aa6a32f70fa2282fe18f7f7c23b6b3a41165c72b1dd4bdb415
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
                      Diff: 1168 upgraded, 11 downgraded, 96 removed, 29 added

● fedora:fedora/35/x86_64/silverblue
                   Version: 35.20220227.0 (2022-02-27T03:23:20Z)
                    Commit: e16355f75eed8f696622850772abcf61be6e8deab1def2402f7bebc8faa95890
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F

So something must be different on your system at least.

Could you include the output from rpm-ostree --version?

hkaancaliskan commented 2 years ago

@juhp I'm using dual boot with Windows 11 on the same disk with secure boot enabled and virtualization is disabled on bios. That's all i can think of the differences 🤷 I can provide any other info you need.

Edit: also Silverblue is luks encrypted.

Edit 2: I'm redownloading 36, will provide rpm-ostree --version

hkaancaliskan commented 2 years ago

@juhp

[kaan@fedora ~]$ rpm-ostree --version 
rpm-ostree:
 Version: '2022.2'
 Git: 8a9c569d34588d8b63f06a5b8b3acf457f897d69
 Features:
  - rust
  - compose
  - fedora-integration

● fedora:fedora/36/x86_64/silverblue
                   Version: 36.20220227.n.0 (2022-02-27T08:03:47Z)
                BaseCommit: bc865956472db6aa6a32f70fa2282fe18f7f7c23b6b3a41165c72b1dd4bdb415
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
       RemovedBasePackages: firefox 96.0-1.fc36 gnome-tour 41~rc-2.fc36

travier commented 2 years ago

Can you check with no removed packages? Maybe we are missing a dependency on something that is pulled by Firefox.

hkaancaliskan commented 2 years ago

[kaan@fedora ~]$ rpm-ostree status 
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● fedora:fedora/36/x86_64/silverblue
                   Version: 36.20220227.n.0 (2022-02-27T08:03:47Z)
                    Commit: bc865956472db6aa6a32f70fa2282fe18f7f7c23b6b3a41165c72b1dd4bdb415
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4

  fedora:fedora/35/x86_64/silverblue
                   Version: 35.20220227.0 (2022-02-27T03:23:20Z)
                BaseCommit: e16355f75eed8f696622850772abcf61be6e8deab1def2402f7bebc8faa95890
              GPGSignature: Valid signature by 787EA6AE1147EEE56C40B30CDB4639719867C58F
       RemovedBasePackages: firefox 97.0.1-2.fc35 gnome-tour 41~rc-1.fc35
                    Pinned: yes

[kaan@fedora ~]$ rpm-ostree upgrade 
note: automatic updates (stage) are enabled
error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`.

@travier Still the same :/

I've also tried with disabling automatic updates, not fixed.

And as a last hope i tried re-enabling cpu virtualization from bios but that didn't helped.

hkaancaliskan commented 2 years ago

Could you try to reproduce with luks encrypted (/boot not encrypted) and dual boot windows 11? Preferably both at the same time. There is just this one left as a difference from yours :/

travier commented 2 years ago

I can not reproduce in a VM on F36 with LUKS (but no Windows):

[silverblue@fedora ~]$ rpm-ostree status
State: idle
Deployments:
● fedora:fedora/36/x86_64/silverblue
                   Version: 36.20220228.n.0 (2022-02-28T07:56:06Z)
                    Commit: 081d8f399213f1bc1c77b1ec6aa1500c9423cb3beab79465b900409b6a1fc128
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4

  fedora:fedora/36/x86_64/silverblue
                   Version: 36.20220214.n.1 (2022-02-14T16:48:21Z)
                    Commit: 29723bb326809b94adea37cadefe80b98dcf3ee7cfa785a764f5d25dce745549
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
[silverblue@fedora ~]$ lsblk 
NAME                                          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sr0                                            11:0    1 1024M  0 rom   
zram0                                         251:0    0  3.8G  0 disk  [SWAP]
vda                                           252:0    0   20G  0 disk  
├─vda1                                        252:1    0    1G  0 part  /boot
└─vda2                                        252:2    0   19G  0 part  
  └─luks-088e7715-4cec-4992-bb20-74cb6edb3bcc 253:0    0   19G  0 crypt /sysroot/ostree/deploy/fedora/var/home
                                                                        /var/home
                                                                        /var
                                                                        /usr
                                                                        /
                                                                        /sysroot
[silverblue@fedora ~]$ sudo rpm-ostree update
[sudo] password for silverblue: 
2 metadata, 0 content objects fetched; 788 B transferred in 1 seconds; 0 bytes content written
No upgrade available.

hkaancaliskan commented 2 years ago

okeeey, what we're going to do now?

jlebon commented 2 years ago

What does sudo ostree admin config-diff say?

hkaancaliskan commented 2 years ago

@jlebon

[kaan@fedora ~]$ sudo ostree admin config-diff
M    adjtime
M    group
M    passwd
M    rpm-ostreed.conf
M    gshadow
M    shadow
M    machine-id
M    subgid
M    subuid
M    authselect/authselect.conf
M    authselect/dconf-db
M    authselect/dconf-locks
M    authselect/fingerprint-auth
M    authselect/smartcard-auth
M    authselect/postlogin
M    authselect/password-auth
M    authselect/system-auth
M    authselect/nsswitch.conf
M    bluetooth/main.conf
M    cups/subscriptions.conf
M    dconf/db/gdm
M    dconf/db/site
M    dconf/db/local
M    dconf/db/distro
M    gdm/custom.conf
M    udev/hwdb.bin
M    yum.repos.d/fedora-cisco-openh264.repo
M    yum.repos.d/fedora-updates-archive.repo
M    yum.repos.d/fedora-modular.repo
D    systemd/system/network-online.target.wants
A    NetworkManager/system-connections/enp2s0f1.nmconnection
A    NetworkManager/system-connections/eda_apart_2 .nmconnection
A    NetworkManager/system-connections/Galaxy A52.nmconnection
A    NetworkManager/system-connections/eda_apart_2.kat_2.nmconnection
A    X11/xorg.conf.d/00-keyboard.conf
A    audit/audit.rules
A    chromium/policies/managed/00_gssapi.json
A    cups/subscriptions.conf.O
A    default/grub
A    opt/chrome/policies/managed/00_gssapi.json
A    opt/chrome/managed
A    sysconfig/anaconda
A    sysconfig/network
A    systemd/system/timers.target.wants/rpm-ostreed-automatic.timer
A    systemd/system/default.target
A    systemd/system.control
A    systemd/system.control/user.slice.d
A    systemd/system.control/user.slice.d/50-MemoryMin.conf
A    systemd/system.control/user.slice.d/50-MemoryLow.conf
A    systemd/system.control/user-975.slice.d
A    systemd/system.control/user-975.slice.d/50-MemoryMin.conf
A    systemd/system.control/user-975.slice.d/50-MemoryLow.conf
A    systemd/system.control/user-975.slice.d/50-CPUWeight.conf
A    systemd/system.control/user-975.slice.d/50-IOWeight.conf
A    systemd/system.control/user@975.service.d
A    systemd/system.control/user@975.service.d/50-MemoryMin.conf
A    systemd/system.control/user@975.service.d/50-MemoryLow.conf
A    systemd/system.control/user@975.service.d/50-CPUWeight.conf
A    systemd/system.control/user@975.service.d/50-IOWeight.conf
A    systemd/system.control/user-1000.slice.d
A    systemd/system.control/user-1000.slice.d/50-MemoryMin.conf
A    systemd/system.control/user-1000.slice.d/50-MemoryLow.conf
A    systemd/system.control/user-1000.slice.d/50-CPUWeight.conf
A    systemd/system.control/user-1000.slice.d/50-IOWeight.conf
A    systemd/system.control/user@1000.service.d
A    systemd/system.control/user@1000.service.d/50-MemoryMin.conf
A    systemd/system.control/user@1000.service.d/50-MemoryLow.conf
A    systemd/system.control/user@1000.service.d/50-CPUWeight.conf
A    systemd/system.control/user@1000.service.d/50-IOWeight.conf
A    systemd/system.control/user-42.slice.d
A    systemd/system.control/user-42.slice.d/50-MemoryMin.conf
A    systemd/system.control/user-42.slice.d/50-MemoryLow.conf
A    systemd/system.control/user-42.slice.d/50-CPUWeight.conf
A    systemd/system.control/user-42.slice.d/50-IOWeight.conf
A    systemd/system.control/user@42.service.d
A    systemd/system.control/user@42.service.d/50-MemoryMin.conf
A    systemd/system.control/user@42.service.d/50-MemoryLow.conf
A    systemd/system.control/user@42.service.d/50-CPUWeight.conf
A    systemd/system.control/user@42.service.d/50-IOWeight.conf
A    yum.repos.d/rpmfusion-free-updates.repo
A    crypttab
A    localtime
A    vconsole.conf
A    hostname
A    .pwd.lock
A    mtab
A    resolv.conf
A    subuid-
A    subgid-
A    cni
A    cni/net.d
A    cni/net.d/87-podman.conflist
A    fstab
A    locale.conf
A    group-
A    gshadow-
A    passwd-
A    shadow-
A    .updated

hkaancaliskan commented 2 years ago

As a side note: I've installed silverblue at 17 February so it's not old. There shouldn't be weird upgrade artefacts from the past. Is there a way to verify system files to ensure files are not broken and same with server side?

travier commented 2 years ago

You can try ostree fsck: https://www.mankier.com/1/ostree-fsck

hkaancaliskan commented 2 years ago

[kaan@fedora ~]$ sudo ostree fsck -a
[sudo] password for kaan: 
Validating refs...
Validating refs in collections...
Enumerating objects...
Verifying content integrity of 7 commit objects...
fsck objects (149338/149338) [=============] 100%
1 partial commits not verified
object fsck of 7 commits completed successfully - no errors found.

Looks good

This is on 35. I'll check with 36 too.

Edit: Same result on 36.

hkaancaliskan commented 2 years ago

I don't know if it helps any but here's the full log of system via journalctl -xb. log.txt

hkaancaliskan commented 2 years ago

Core dump also happens when i try to curl https://google.com -v (Side note: there is no issue with wget)

[kaan@fedora ~]$ curl https://google.com -v
*   Trying 216.239.38.120:443...
* Connected to google.com (216.239.38.120) port 443 (#0)
Parçalama arızası (çekirdek döküldü) //means "segmentation fault (core dumped)" in turkish

And when i take a look on coredumpctl info, i see exact same log. (btw i've rebased to kinoite for try, it also didn't worked and this log is from that)

[kaan@fedora ~]$ sudo coredumpctl info
           PID: 7239 (curl)
           UID: 1000 (kaan)
           GID: 1000 (kaan)
        Signal: 11 (SEGV)
     Timestamp: Sat 2022-03-05 22:53:18 +03 (1min 24s ago)
  Command Line: curl https://google.com -v
    Executable: /usr/bin/curl
 Control Group: /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.kde.konsole-f5794a71c37848faa87aa8e8e6f46d55.scope
          Unit: user@1000.service
     User Unit: app-org.kde.konsole-f5794a71c37848faa87aa8e8e6f46d55.scope
         Slice: user-1000.slice
     Owner UID: 1000 (kaan)
       Boot ID: c97199c977fd4f0d8d54b596f507872e
    Machine ID: 67d340ffbfe74a2cb742e1138a063f33
      Hostname: fedora
       Storage: /var/lib/systemd/coredump/core.curl.1000.c97199c977fd4f0d8d54b596f507872e.7239.1646509998000000.zst (present)
     Disk Size: 255.9K
       Message: Process 7239 (curl) of user 1000 dumped core.

                Module linux-vdso.so.1 with build-id 84c06a63053a82c1587d88212adfc75736d4e885
                Module libm.so.6 with build-id 2874fc1583dec9958293fcb9b5f6453037eeb6bc
                Module libnss_resolve.so.2 with build-id 74cb82260f0e39b720154cc57852ee07d3aa612b
                Metadata for module libnss_resolve.so.2 owned by FDO found: {
                        "type" : "rpm",
                        "name" : "systemd",
                        "version" : "250.3-4.fc36",
                        "architecture" : "x86_64",
                        "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                }

                Module libnss_mdns4_minimal.so.2 with build-id d2f6978d873a3f16c4a4da32c136ebe995550eff
                Metadata for module libnss_mdns4_minimal.so.2 owned by FDO found: {
                        "type" : "rpm",
                        "name" : "nss-mdns",
                        "version" : "0.15.1-4.fc36",
                        "architecture" : "x86_64",
                        "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                }

                Module libgcc_s.so.1 with build-id ea09d824cf066135cbeb4c5f6427129729d5b9c4
                Module libnss_myhostname.so.2 with build-id 347b028b15782fddc705758aa30a16dd07e43195
                Metadata for module libnss_myhostname.so.2 owned by FDO found: {
                        "type" : "rpm",
                        "name" : "systemd",
                        "version" : "250.3-4.fc36",
                        "architecture" : "x86_64",
                        "osCpe" : "cpe:/o:fedoraproject:fedora:36"
                }

                Module libcrypt.so.2 with build-id 9924d3a82ca8fa6b94bc574a6520bcf84ccf831f
                Stack trace of thread 7239:
                #0  0x00007ff77838b184 EVP_CIPHER_get_nid (libcrypto.so.3 + 0x185184)
                #1  0x00007ff778395630 EVP_CIPHER_get_type (libcrypto.so.3 + 0x18f630)
                #2  0x00007ff7783bdf4f get_legacy_cipher_names (libcrypto.so.3 + 0x1b7f4f)
                #3  0x00007ff7783ac637 OPENSSL_LH_doall_arg (libcrypto.so.3 + 0x1a6637)
                #4  0x00007ff7783cc82b OBJ_NAME_do_all (libcrypto.so.3 + 0x1c682b)
                #5  0x00007ff7783bda9a ossl_namemap_stored (libcrypto.so.3 + 0x1b7a9a)
                #6  0x00007ff778493691 inner_evp_generic_fetch.constprop.0 (libcrypto.so.3 + 0x28d691)
                #7  0x00007ff778395f64 EVP_RAND_fetch (libcrypto.so.3 + 0x18ff64)
                #8  0x00007ff7783e88d3 RAND_get0_primary (libcrypto.so.3 + 0x1e28d3)
                #9  0x00007ff7783e8ae7 RAND_status (libcrypto.so.3 + 0x1e2ae7)
                #10 0x00007ff778a2b24e ossl_connect_step1 (libcurl.so.4 + 0x6d24e)
                #11 0x00007ff778a32d87 ossl_connect_common (libcurl.so.4 + 0x74d87)
                #12 0x00007ff778a34988 Curl_ssl_connect_nonblocking (libcurl.so.4 + 0x76988)
                #13 0x00007ff7789eadc6 Curl_http_connect (libcurl.so.4 + 0x2cdc6)
                #14 0x00007ff778a07e0f multi_runsingle (libcurl.so.4 + 0x49e0f)
                #15 0x00007ff778a0ae3e curl_multi_perform (libcurl.so.4 + 0x4ce3e)
                #16 0x00007ff7789e1da3 curl_easy_perform (libcurl.so.4 + 0x23da3)
                #17 0x0000557bc9ef1b73 main (curl + 0xdb73)
                #18 0x00007ff7787cf590 __libc_start_call_main (libc.so.6 + 0x2d590)
                #19 0x00007ff7787cf649 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2d649)
                #20 0x0000557bc9ef2075 _start (curl + 0xe075)
                ELF object binary architecture: AMD x86-64

hkaancaliskan commented 2 years ago

Is there a possibility that this problem is related to https://github.com/openssl/openssl/pull/16963#issuecomment-961584695 because that's the only commit references to nid which version 3.0.1 has but 3.0.0 doesn't.

hkaancaliskan commented 2 years ago

I've tried with clean install alongside windows 11. It didn't fixed.

My laptop: i5 8250U UHD620+MX130 12GB RAM

Good luck 🤞

Side note: Silverblue is encrypted with luks, maybe this is related to cryptsetup?

hkaancaliskan commented 2 years ago

@travier I've installed 35 on gnome-boxes and right after installation upgraded to 36. Same error. Luks encrypted, 20gb disk, 3gb ram I'm using the iso from https://silverblue.fedoraproject.org/download x86_64version

Edit: I've also setup same VM without luks and still same. I've also tried with another wifi and mobile data, same result.

travier commented 2 years ago

At this point, as we can not reproduce it, I suggest that you install the debuginfo packages and get a backtrace via gdb so that we can try to figure out what happens. Maybe this should be moved to rpm-ostree.

hkaancaliskan commented 2 years ago

As i can already reproduce in VM, ofc i can help with any info you need

How can i install debuginfo packages? Is there any guide for Silverblue?

Edit: Btw curl working fine inside toolbox.

tpopela commented 2 years ago

As i can already reproduce in VM, ofc i can help with any info you need

How can i install debuginfo packages? Is there any guide for Silverblue?

Use https://gist.github.com/tpopela/b9a390a6a15a21b74b1ea3d4b3ee9557 to unlock the current deployment and install DNF into it. Then reproduce the crash and use coredumpctl gdb to open the gdb and it will download debuginfos for you. Then get the backtrace with t a a bt.

hkaancaliskan commented 2 years ago

Sorry for the delay because i can't download them, just throws error :) Downloading dependencies manually is hard because i don't know which dependencies does it need. I tried downloading them on 35 and upgrading to 36 but they're going away, idk why. Offline installing of debuginfos not working, i can't seem to find a way.

It's harder than i think :) I'll try when i have more free time.

Edit: Actually one thing can solve my problems, how can i install debuginfos of rpm-ostree.fc36 etc from silverblue 35. I think i need to add repos of 36 but i'm not able to find them.

hkaancaliskan commented 2 years ago

I've downloaded Silverblue 36 iso from https://dl.fedoraproject.org/pub/fedora/linux/development/36/Silverblue/x86_64/iso/ and it's working perfectly fine. Problem is just with upgrade from 35 to 36. I'm unable to install debuginfos etc to debug the problem because when internet is missing, that's very hard. I can clean install when Silverblue 36 released. If no one else has this problem, i think it's okey to ignore/close the issue.

Thanks!

hkaancaliskan commented 2 years ago

I was wrong. I've installed that SB36 iso on VM and that's worked, I thought it's fixed. Now I've tried to install on my laptop and same error comes back. Sorry guys, unable to debug the issue :(

travier commented 2 years ago

Looks like that curl error could be reproduced on classic Fedora where it would be easier for you to install packages maybe?

hkaancaliskan commented 2 years ago

I've installed classic Fedora 36 - 20220325 iso and now it's not coredumping but throws another error.

resim

~~How can i debug this?~~

I've found another one who has the same issue from Turkey on a forum https://www.technopat.net/sosyal/konu/fedora-rawhide-curl-bozuk.1907403/#post-14950151 ~~Maybe this issue is related to ISS' of Turkey :/~~

Edit: Workaround for this one is LC_ALL=C curl https://google.com -v with LC_ALL=C curl and dnf works fine for classic Fedora. https://www.technopat.net/sosyal/konu/fedora-rawhide-curl-bozuk-decode-error.1907403/post-14955660

But on silverblue, this workaround fixes curl but not rpm-ostree. Rpm-ostree still throws bus owner changed error and same coredump.

Edit 2: At least curl works fine now on silverblue. I will try to install debug packages again.

hkaancaliskan commented 2 years ago

I'm not able to debug the rpm-ostree but I've reinstalled silverblue as English and problem is gone. Everything working fine (tried also on my laptop).

@travier Problem is just with Turkish. Could you please try to reproduce?

hkaancaliskan commented 2 years ago

As a side note: When you install silverblue with Turkish and luks enabled, installation fails while partitioning. There is no problem on English.

queeup commented 2 years ago

Same problem here. Turkish language fedora silverblue rawhide. It happens to dnf and curl on classic fedora 36 beta too.

$ rpm-ostree upgrade
error: While pulling fedora/rawhide/x86_64/silverblue: While fetching mirrorlist 
'https://ostree.fedoraproject.org/mirrorlist': While fetching https://ostree.fedoraproject.org/mirrorlist:
[35] SSL connect error

$ curl -L https://ostree.fedoraproject.org/mirrorlist
curl: (35) error:03000072:digital envelope routines::decode error

$ LC_ALL=C curl -L https://ostree.fedoraproject.org/mirrorlist
https://d2uk5hbyrobdzx.cloudfront.net/

$ locale
LANG=tr_TR.UTF-8
LC_CTYPE="tr_TR.UTF-8"
LC_NUMERIC="tr_TR.UTF-8"
LC_TIME="tr_TR.UTF-8"
LC_COLLATE="tr_TR.UTF-8"
LC_MONETARY="tr_TR.UTF-8"
LC_MESSAGES="tr_TR.UTF-8"
LC_PAPER="tr_TR.UTF-8"
LC_NAME="tr_TR.UTF-8"
LC_ADDRESS="tr_TR.UTF-8"
LC_TELEPHONE="tr_TR.UTF-8"
LC_MEASUREMENT="tr_TR.UTF-8"
LC_IDENTIFICATION="tr_TR.UTF-8"
LC_ALL=
[queeup@fedora ~]$ cat /etc/os
os-release  ostree/     

$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="Rawhide.20220328.n.0 (Silverblue Prerelease)"
ID=fedora
VERSION_ID=37
VERSION_CODENAME=""
PLATFORM_ID="platform:f37"
PRETTY_NAME="Fedora Linux Rawhide.20220328.n.0 (Silverblue Prerelease)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:37"
HOME_URL="https://silverblue.fedoraproject.org"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-silverblue/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://github.com/fedora-silverblue/issue-tracker/issues"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=rawhide
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=rawhide
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Silverblue"
VARIANT_ID=silverblue
OSTREE_VERSION='Rawhide.20220328.n.0'

juhp commented 2 years ago

Turkish language fedora silverblue rawhide. It happens to dnf and curl on classic fedora 36 beta too.

This is the second mention of Turkish locale: quite surprising.

$ LC_ALL=C curl -L https://ostree.fedoraproject.org/mirrorlist

(Anyway this is not a SB or ostree issue)

juhp commented 2 years ago

I can reproduce in Rawhide toolbox (openssl-libs-3.0.0-1.fc36):

$ sudo dnf install glibc-langpack-tr
$ LANG=tr_TR.utf8 curl -L https://google.com
Segmentation fault (core dumped)

and in a F36 toolbox:

$ rpm -q openssl-libs 
openssl-libs-3.0.2-1.fc36.x86_64
$ LANG=tr_TR.utf8 curl -L https://google.com
curl: (35) error:03000072:digital envelope routines::decode error

I can also reproduce in my SB36 laptop (with only glibc-all-langpacks).

queeup commented 2 years ago

Does the locale make a difference? It seems suspicious if this is only happening in Turkey... Can you please try prefixing your commands with LANG=C.utf8 or LANG=en_US.utf8 say?

$ LANG=C.utf8 curl -L https://ostree.fedoraproject.org/mirrorlist
https://d2uk5hbyrobdzx.cloudfront.net/
$ LANG=en_US.utf8 curl -L https://ostree.fedoraproject.org/mirrorlist
https://d2uk5hbyrobdzx.cloudfront.net/

It works well on other locales.

juhp commented 2 years ago

I went ahead and opened https://bugzilla.redhat.com/show_bug.cgi?id=2071343

But it might well be good to report it to upstream too.

queeup commented 2 years ago

This is quite a critic bug. dnf and rpm-ostree are unusable for general user on Turkish language.

travier commented 2 years ago

As this is not strictly speaking a Silverblue bug and there is work upstream and a BZ to follow I'll go ahead and close this one. Thanks for the investigation.

juhp commented 2 years ago

This is fixed now by openssl-3.0.2-3.fc36 which was pushed stable 3 days ago.

fedora-silverblue / issue-tracker

[BUG] rpm-ostree core dumps on 36 #241