search

tf-encrypted / moose

Secure distributed dataflow framework for encrypted machine learning and data processing
Apache License 2.0
57 stars 15 forks source link

Make `msgpack` optional #1026

Closed mortendahl closed 2 years ago

mortendahl commented 2 years ago

This PR puts MsgPack support before a msgpack feature, saving ~850k lines of LLVM code.

It also adds a new full feature that enables everything, allowing us to be more restricted wrt default features.

Before merging we should validate that the benefits are worth having the extra feature. It seems that MsgPack is still needed by Elk and PyMoose, meaning we're essentially only talking about sightly improving the development process.

Closes https://github.com/tf-encrypted/runtime/issues/1020

mortendahl commented 2 years ago

From within moose/:

cargo llvm-lines --no-default-features --features 'compile,sync_execute,async_execute' | head -20

Before:

  Lines            Copies         Function name
  -----            ------         -------------
  19442302 (100%)  634914 (100%)  (TOTAL)
    719496 (3.7%)   12718 (2.0%)  std::panicking::try
    533904 (2.7%)    3632 (0.6%)  tokio::runtime::task::list::OwnedTasks<S>::bind
    456395 (2.3%)   17137 (2.7%)  core::ops::function::FnOnce::call_once
    419694 (2.2%)   12718 (2.0%)  std::panicking::try::do_catch
    382676 (2.0%)    6524 (1.0%)  alloc::alloc::box_free
    362087 (1.9%)     261 (0.0%)  <&mut rmp_serde::decode::Deserializer<R,C> as serde::de::Deserializer>::deserialize_any
    321612 (1.7%)   12718 (2.0%)  std::panicking::try::do_call
    308720 (1.6%)    3632 (0.6%)  tokio::runtime::task::harness::Harness<T,S>::poll_inner
    295198 (1.5%)   14668 (2.3%)  core::ptr::read
    292470 (1.5%)    7266 (1.1%)  tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
    263320 (1.4%)    3632 (0.6%)  tokio::runtime::task::raw::RawTask::new
    257872 (1.3%)    1816 (0.3%)  tokio::runtime::thread_pool::worker::Shared::bind_new_task
    248792 (1.3%)    1816 (0.3%)  tokio::runtime::task::harness::poll_future
    236080 (1.2%)    1816 (0.3%)  tokio::runtime::basic_scheduler::Spawner::spawn
    200641 (1.0%)   18580 (2.9%)  core::mem::manually_drop::ManuallyDrop<T>::new
    196128 (1.0%)    3632 (0.6%)  tokio::runtime::task::harness::Harness<T,S>::release
    196128 (1.0%)    3632 (0.6%)  tokio::runtime::task::new_task

After:

  Lines            Copies         Function name
  -----            ------         -------------
  18590526 (100%)  623434 (100%)  (TOTAL)
    719496 (3.9%)   12718 (2.0%)  std::panicking::try
    533904 (2.9%)    3632 (0.6%)  tokio::runtime::task::list::OwnedTasks<S>::bind
    454615 (2.4%)   16991 (2.7%)  core::ops::function::FnOnce::call_once
    419694 (2.3%)   12718 (2.0%)  std::panicking::try::do_catch
    382570 (2.1%)    6522 (1.0%)  alloc::alloc::box_free
    321612 (1.7%)   12718 (2.0%)  std::panicking::try::do_call
    308720 (1.7%)    3632 (0.6%)  tokio::runtime::task::harness::Harness<T,S>::poll_inner
    295095 (1.6%)   14664 (2.4%)  core::ptr::read
    292470 (1.6%)    7266 (1.2%)  tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
    263320 (1.4%)    3632 (0.6%)  tokio::runtime::task::raw::RawTask::new
    257872 (1.4%)    1816 (0.3%)  tokio::runtime::thread_pool::worker::Shared::bind_new_task
    248792 (1.3%)    1816 (0.3%)  tokio::runtime::task::harness::poll_future
    236080 (1.3%)    1816 (0.3%)  tokio::runtime::basic_scheduler::Spawner::spawn
    200625 (1.1%)   18578 (3.0%)  core::mem::manually_drop::ManuallyDrop<T>::new
    196128 (1.1%)    3632 (0.6%)  tokio::runtime::task::harness::Harness<T,S>::release
    196128 (1.1%)    3632 (0.6%)  tokio::runtime::task::new_task
    192496 (1.0%)    3632 (0.6%)  tokio::runtime::task::harness::Harness<T,S>::try_read_output
mortendahl commented 2 years ago

On the downside, in terms of compile time it doesn't make a significant difference based on a rough test, and it only drops libmoose.rlib from 258MB to 246MB.