search

ostreedev / ostree

Operating system and container binary deployment and upgrades
https://ostreedev.github.io/ostree/
Other
1.26k stars 291 forks source link

Support --fsync=syncfs/ostree_repo_set_fsync_type() #1184

Open cgwalters opened 6 years ago

cgwalters commented 6 years ago

Updated proposal:

Add

typedef enum {
  OSTREE_REPO_SYNC_NONE,
  OSTREE_REPO_SYNC_FILESYSTEM,
  OSTREE_REPO_SYNC_INDIVIDUAL
} OstreeRepoSyncType;

ostree_repo_set_sync_type (OstreeRepo *repo, OstreeRepoSyncType synctype);

And expose this as e.g. `ostree commit --fsync=syncfs` too.

We've had lots of issues with this over time...anyways I was debugging a segfault from what I thought was some outstanding work and saw the stack below.

Basically we're doing fsync() on object commit even if we're in a txn by default still. I think this is pretty broken. Actually embarassing since we've done all this work on using syncfs() and the per-boot-id staging directory etc. that is much less useful if we're doing individual syncs.

        Signal: 11 (SEGV)
     Timestamp: Mon 2017-09-18 15:17:02 UTC (1min 56s ago)
  Command Line: ostree --repo=mnt/repo pull-local /ostree/repo f8595ef11ece381bea837d470fd5baf1a2b2301d8300029bd021d4b9b3a00da7
    Executable: /usr/bin/ostree
 Control Group: /user.slice/user-1000.slice/session-4.scope
          Unit: session-4.scope
         Slice: user-1000.slice
       Session: 4
     Owner UID: 1000 (vagrant)
       Boot ID: a2d1e14193434c3aada9135ed5694f4c
    Machine ID: 60cab8c8a6b84d178c3f175dfcf30e1b
      Hostname: localhost.localdomain
       Storage: /var/lib/systemd/coredump/core.ostree.0.a2d1e14193434c3aada9135ed5694f4c.3106.1505747822000000.lz4
       Message: Process 3106 (ostree) of user 0 dumped core.

                Stack trace of thread 3109:
                #0  0x00007f2dbd9ef88b import_via_hardlink_is_possible (libostree-1.so.1)
                #1  0x00007f2dbd9ef934 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #2  0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #3  0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
                #4  0x00007f2dbc939086 g_task_thread_pool_thread (libgio-2.0.so.0)
                #5  0x00007f2dbc3bbf00 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                #6  0x00007f2dbc3bb536 g_thread_proxy (libglib-2.0.so.0)
                #7  0x00007f2dbbe7f36d start_thread (libpthread.so.0)
                #8  0x00007f2dbbbb7bbf __clone (libc.so.6)

                Stack trace of thread 3107:
                #0  0x00007f2dbbbabacd poll (libc.so.6)
                #1  0x00007f2dbc394569 g_main_context_iterate.isra.25 (libglib-2.0.so.0)
                #2  0x00007f2dbc39467c g_main_context_iteration (libglib-2.0.so.0)
                #3  0x00007f2dbc3946c1 glib_worker_main (libglib-2.0.so.0)
                #4  0x00007f2dbc3bb536 g_thread_proxy (libglib-2.0.so.0)
                #5  0x00007f2dbbe7f36d start_thread (libpthread.so.0)
                #6  0x00007f2dbbbb7bbf __clone (libc.so.6)

                Stack trace of thread 3111:
                #0  0x00007f2dbbe8913d __write (libpthread.so.0)
                #1  0x00007f2dbda48c46 glnx_loop_write (libostree-1.so.1)
                #2  0x00007f2dbd9f8d67 create_regular_tmpfile_linkable_with_content (libostree-1.so.1)
                #3  0x00007f2dbd9f9344 write_content_object (libostree-1.so.1)
                #4  0x00007f2dbd9fc32f ostree_repo_write_content (libostree-1.so.1)
                #5  0x00007f2dbd9fc278 ostree_repo_write_content_trusted (libostree-1.so.1)
                #6  0x00007f2dbd9efb88 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #7  0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #8  0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
                #9  0x00007f2dbc939086 g_task_thread_pool_thread (libgio-2.0.so.0)
                #10 0x00007f2dbc3bbf00 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                #11 0x00007f2dbc3bb536 g_thread_proxy (libglib-2.0.so.0)
                #12 0x00007f2dbbe7f36d start_thread (libpthread.so.0)
                #13 0x00007f2dbbbb7bbf __clone (libc.so.6)

                Stack trace of thread 3116:
                #0  0x00007f2dbbe8919d read (libpthread.so.0)
                #1  0x00007f2dbc9a6a05 g_unix_input_stream_read (libgio-2.0.so.0)
                #2  0x00007f2dbc910017 g_input_stream_read (libgio-2.0.so.0)
                #3  0x00007f2dbd9e2c07 ostree_chain_input_stream_read (libostree-1.so.1)
                #4  0x00007f2dbc910017 g_input_stream_read (libgio-2.0.so.0)
                #5  0x00007f2dbd9f8d06 create_regular_tmpfile_linkable_with_content (libostree-1.so.1)
                #6  0x00007f2dbd9f9344 write_content_object (libostree-1.so.1)
                #7  0x00007f2dbd9fc32f ostree_repo_write_content (libostree-1.so.1)
                #8  0x00007f2dbd9fc278 ostree_repo_write_content_trusted (libostree-1.so.1)
                #9  0x00007f2dbd9efb88 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #10 0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #11 0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
                #12 0x00007f2dbc939086 g_task_thread_pool_thread (libgio-2.0.so.0)
                #13 0x00007f2dbc3bbf00 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                #14 0x00007f2dbc3bb536 g_thread_proxy (libglib-2.0.so.0)
                #15 0x00007f2dbbe7f36d start_thread (libpthread.so.0)
                #16 0x00007f2dbbbb7bbf __clone (libc.so.6)

                Stack trace of thread 3106:
                #0  0x00007f2db790f8d4 _fini (liblz4.so.1)
                #1  0x00007f2dbbae1c68 __run_exit_handlers (libc.so.6)
                #2  0x00007f2dbbae1cba exit (libc.so.6)
                #3  0x00007f2dbbac7511 __libc_start_main (libc.so.6)
                #4  0x000000000040abfa _start (ostree)

                Stack trace of thread 3114:
                #0  0x00007f2dbbe8919d read (libpthread.so.0)
                #1  0x00007f2dbc9a6a05 g_unix_input_stream_read (libgio-2.0.so.0)
                #2  0x00007f2dbc910017 g_input_stream_read (libgio-2.0.so.0)
                #3  0x00007f2dbd9e2c07 ostree_chain_input_stream_read (libostree-1.so.1)
                #4  0x00007f2dbc910017 g_input_stream_read (libgio-2.0.so.0)
                #5  0x00007f2dbd9f8d06 create_regular_tmpfile_linkable_with_content (libostree-1.so.1)
                #6  0x00007f2dbd9f9344 write_content_object (libostree-1.so.1)
                #7  0x00007f2dbd9fc32f ostree_repo_write_content (libostree-1.so.1)
                #8  0x00007f2dbd9fc278 ostree_repo_write_content_trusted (libostree-1.so.1)
                #9  0x00007f2dbd9efb88 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #10 0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #11 0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
                #12 0x00007f2dbc939086 g_task_thread_pool_thread (libgio-2.0.so.0)
                #13 0x00007f2dbc3bbf00 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                #14 0x00007f2dbc3bb536 g_thread_proxy (libglib-2.0.so.0)
                #15 0x00007f2dbbe7f36d start_thread (libpthread.so.0)
                #16 0x00007f2dbbbb7bbf __clone (libc.so.6)

                Stack trace of thread 3115:
                #0  0x00007f2dbbe8973d fsync (libpthread.so.0)
                #1  0x00007f2dbd9f85f3 commit_loose_regfile_object (libostree-1.so.1)
                #2  0x00007f2dbd9f99ca write_content_object (libostree-1.so.1)
                #3  0x00007f2dbd9fc32f ostree_repo_write_content (libostree-1.so.1)
                #4  0x00007f2dbd9fc278 ostree_repo_write_content_trusted (libostree-1.so.1)
                #5  0x00007f2dbd9efb88 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #6  0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #7  0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
                #8  0x00007f2dbc939086 g_task_thread_pool_thread (libgio-2.0.so.0)
                #9  0x00007f2dbc3bbf00 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                #10 0x00007f2dbc3bb536 g_thread_proxy (libglib-2.0.so.0)
                #11 0x00007f2dbbe7f36d start_thread (libpthread.so.0)
                #12 0x00007f2dbbbb7bbf __clone (libc.so.6)

                Stack trace of thread 3117:
                #0  0x00007f2dbbe8973d fsync (libpthread.so.0)
                #1  0x00007f2dbd9f85f3 commit_loose_regfile_object (libostree-1.so.1)
                #2  0x00007f2dbd9f99ca write_content_object (libostree-1.so.1)
                #3  0x00007f2dbd9fc32f ostree_repo_write_content (libostree-1.so.1)
                #4  0x00007f2dbd9fc278 ostree_repo_write_content_trusted (libostree-1.so.1)
                #5  0x00007f2dbd9efb88 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #6  0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #7  0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
                #8  0x00007f2dbc939086 g_task_thread_pool_thread (libgio-2.0.so.0)
                #9  0x00007f2dbc3bbf00 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                #10 0x00007f2dbc3bb536 g_thread_proxy (libglib-2.0.so.0)
                #11 0x00007f2dbbe7f36d start_thread (libpthread.so.0)
                #12 0x00007f2dbbbb7bbf __clone (libc.so.6)

                Stack trace of thread 3112:
                #0  0x00007f2dbbe8973d fsync (libpthread.so.0)
                #1  0x00007f2dbd9f85f3 commit_loose_regfile_object (libostree-1.so.1)
                #2  0x00007f2dbd9f99ca write_content_object (libostree-1.so.1)
                #3  0x00007f2dbd9fc32f ostree_repo_write_content (libostree-1.so.1)
                #4  0x00007f2dbd9fc278 ostree_repo_write_content_trusted (libostree-1.so.1)
                #5  0x00007f2dbd9efb88 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #6  0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #7  0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
                #7  0x00007f2dbd9fc32f ostree_repo_write_content (libostree-1.so.1)
                #8  0x00007f2dbd9fc278 ostree_repo_write_content_trusted (libostree-1.so.1)
                #9  0x00007f2dbd9efb88 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #10 0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #11 0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
                #12 0x00007f2dbc939086 g_task_thread_pool_thread (libgio-2.0.so.0)
                #13 0x00007f2dbc3bbf00 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                #14 0x00007f2dbc3bb536 g_thread_proxy (libglib-2.0.so.0)
                #15 0x00007f2dbbe7f36d start_thread (libpthread.so.0)
                #16 0x00007f2dbbbb7bbf __clone (libc.so.6)

                Stack trace of thread 3115:
                #0  0x00007f2dbbe8973d fsync (libpthread.so.0)
                #1  0x00007f2dbd9f85f3 commit_loose_regfile_object (libostree-1.so.1)
                #2  0x00007f2dbd9f99ca write_content_object (libostree-1.so.1)
                #3  0x00007f2dbd9fc32f ostree_repo_write_content (libostree-1.so.1)
                #4  0x00007f2dbd9fc278 ostree_repo_write_content_trusted (libostree-1.so.1)
                #5  0x00007f2dbd9efb88 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #6  0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #7  0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
                #8  0x00007f2dbc939086 g_task_thread_pool_thread (libgio-2.0.so.0)
                #9  0x00007f2dbc3bbf00 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                #10 0x00007f2dbc3bb536 g_thread_proxy (libglib-2.0.so.0)
                #11 0x00007f2dbbe7f36d start_thread (libpthread.so.0)
                #12 0x00007f2dbbbb7bbf __clone (libc.so.6)

                Stack trace of thread 3117:
                #0  0x00007f2dbbe8973d fsync (libpthread.so.0)
                #1  0x00007f2dbd9f85f3 commit_loose_regfile_object (libostree-1.so.1)
                #2  0x00007f2dbd9f99ca write_content_object (libostree-1.so.1)
                #3  0x00007f2dbd9fc32f ostree_repo_write_content (libostree-1.so.1)
                #4  0x00007f2dbd9fc278 ostree_repo_write_content_trusted (libostree-1.so.1)
                #5  0x00007f2dbd9efb88 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #6  0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #7  0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
                #8  0x00007f2dbc939086 g_task_thread_pool_thread (libgio-2.0.so.0)
                #9  0x00007f2dbc3bbf00 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                #10 0x00007f2dbc3bb536 g_thread_proxy (libglib-2.0.so.0)
                #11 0x00007f2dbbe7f36d start_thread (libpthread.so.0)
                #12 0x00007f2dbbbb7bbf __clone (libc.so.6)

                Stack trace of thread 3112:
                #0  0x00007f2dbbe8973d fsync (libpthread.so.0)
                #1  0x00007f2dbd9f85f3 commit_loose_regfile_object (libostree-1.so.1)
                #2  0x00007f2dbd9f99ca write_content_object (libostree-1.so.1)
                #3  0x00007f2dbd9fc32f ostree_repo_write_content (libostree-1.so.1)
                #4  0x00007f2dbd9fc278 ostree_repo_write_content_trusted (libostree-1.so.1)
                #5  0x00007f2dbd9efb88 ostree_repo_import_object_from_with_trust (libostree-1.so.1)
                #6  0x00007f2dbda00aeb import_one_local_content_object_sync (libostree-1.so.1)
                #7  0x00007f2dbda00d37 async_import_in_thread (libostree-1.so.1)
cgwalters commented 6 years ago

Without doing a lot of measurement, I suspect we want to have potentially different behavior between pulling to a releng (mirror/dev/test) repo vs a system (or separate flatpak etc.) repo.

A releng repo might be on a large data volume and the cost of a syncfs() is probably much greater than individual fsyncs.