search

anbox / anbox

Anbox is a container-based approach to boot a full Android system on a regular GNU/Linux system
https://anbox.io
GNU General Public License v3.0
8.88k stars 1.15k forks source link

Mount FUSE in unprivileged user namespace #882

Open comword opened 6 years ago

comword commented 6 years ago
  1. Please check that no similar bug is already reported. Have a look on the list of open bugs at https://github.com/anbox/anbox/issues
  2. Make sure you are running the latest version of Anbox before reporting an issue. Update snap to latest: snap refresh --devmode --edge anbox
  3. Make sure you have debug logs enabled: sudo snap set anbox debug.enable=true
  4. Reproduce the error while debug logs enabled.
  5. Run the anbox logs collection utility and attach the tar file. sudo /snap/bin/anbox.collect-bug-info anbox-system-diagnostics-2018-08-26.zip
  6. Please paste the result of anbox system-info below: 
    version: 4
snap-revision: 143
cpu:
arch:  x86
brand: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
features: 
- aes
- sse4_1
- sse4_2
- avx
- avx2
os:
name: Debian GNU/Linux
version: 
snap-based: true
kernel:
version: Linux version 4.17.0-3-amd64 (debian-kernel@lists.debian.org) (gcc version 7.3.0 (Debian 7.3.0-28)) #1 SMP Debian 4.17.17-1 (2018-08-18)
binder: true
ashmem: true
graphics:
egl:
vendor: NVIDIA
version: 1.4
extensions:
  - EGL_EXT_buffer_age
  - EGL_EXT_create_context_robustness
  - EGL_EXT_output_base
  - EGL_EXT_stream_acquire_mode
  - EGL_IMG_context_priority
  - EGL_KHR_config_attribs
  - EGL_KHR_create_context_no_error
  - EGL_KHR_context_flush_control
  - EGL_KHR_create_context
  - EGL_KHR_display_reference
  - EGL_KHR_fence_sync
  - EGL_KHR_get_all_proc_addresses
  - EGL_KHR_partial_update
  - EGL_KHR_swap_buffers_with_damage
  - EGL_KHR_gl_colorspace
  - EGL_KHR_gl_renderbuffer_image
  - EGL_KHR_gl_texture_2D_image
  - EGL_KHR_gl_texture_3D_image
  - EGL_KHR_gl_texture_cubemap_image
  - EGL_KHR_image
  - EGL_KHR_image_base
  - EGL_KHR_image_pixmap
  - EGL_KHR_reusable_sync
  - EGL_KHR_stream
  - EGL_KHR_stream_consumer_gltexture
  - EGL_KHR_stream_cross_process_fd
  - EGL_KHR_stream_fifo
  - EGL_KHR_stream_producer_eglsurface
  - EGL_KHR_surfaceless_context
  - EGL_NV_nvrm_fence_sync
  - EGL_NV_post_sub_buffer
  - EGL_NV_stream_metadata
  - EGL_NV_stream_reset
  - EGL_NV_stream_sync
  - EGL_NV_stream_consumer_gltexture_yuv
  - EGL_NV_stream_attrib
  - EGL_NV_sync
  - EGL_NV_system_time
  - EGL_NV_output_drm_flip_event
gles2:
vendor: NVIDIA Corporation
vendor: OpenGL ES 3.2 NVIDIA 396.37
extensions:
  - GL_EXT_base_instance
  - GL_EXT_blend_func_extended
  - GL_EXT_blend_minmax
  - GL_EXT_buffer_storage
  - GL_EXT_clear_texture
  - GL_EXT_clip_control
  - GL_EXT_clip_cull_distance
  - GL_EXT_color_buffer_float
  - GL_EXT_color_buffer_half_float
  - GL_EXT_conservative_depth
  - GL_EXT_copy_image
  - GL_EXT_debug_label
  - GL_EXT_discard_framebuffer
  - GL_EXT_disjoint_timer_query
  - GL_EXT_draw_buffers_indexed
  - GL_EXT_draw_elements_base_vertex
  - GL_EXT_float_blend
  - GL_EXT_frag_depth
  - GL_EXT_geometry_point_size
  - GL_EXT_geometry_shader
  - GL_EXT_gpu_shader5
  - GL_EXT_map_buffer_range
  - GL_EXT_multi_draw_indirect
  - GL_EXT_multisample_compatibility
  - GL_EXT_occlusion_query_boolean
  - GL_EXT_polygon_offset_clamp
  - GL_EXT_primitive_bounding_box
  - GL_EXT_render_snorm
  - GL_EXT_robustness
  - GL_EXT_separate_shader_objects
  - GL_EXT_shader_group_vote
  - GL_EXT_shader_implicit_conversions
  - GL_EXT_shader_integer_mix
  - GL_EXT_shader_io_blocks
  - GL_EXT_shader_non_constant_global_initializers
  - GL_EXT_shader_texture_lod
  - GL_EXT_shadow_samplers
  - GL_EXT_sparse_texture
  - GL_EXT_sRGB
  - GL_EXT_sRGB_write_control
  - GL_EXT_tessellation_point_size
  - GL_EXT_tessellation_shader
  - GL_EXT_texture_border_clamp
  - GL_EXT_texture_buffer
  - GL_EXT_texture_compression_bptc
  - GL_EXT_texture_compression_dxt1
  - GL_EXT_texture_compression_rgtc
  - GL_EXT_texture_compression_s3tc
  - GL_EXT_texture_cube_map_array
  - GL_EXT_texture_filter_anisotropic
  - GL_EXT_texture_format_BGRA8888
  - GL_EXT_texture_mirror_clamp_to_edge
  - GL_EXT_texture_norm16
  - GL_EXT_texture_rg
  - GL_EXT_texture_sRGB_R8
  - GL_EXT_texture_sRGB_decode
  - GL_EXT_texture_storage
  - GL_EXT_texture_view
  - GL_EXT_draw_transform_feedback
  - GL_EXT_unpack_subimage
  - GL_EXT_window_rectangles
  - GL_KHR_context_flush_control
  - GL_KHR_debug
  - GL_EXT_memory_object
  - GL_EXT_memory_object_fd
  - GL_KHR_parallel_shader_compile
  - GL_KHR_no_error
  - GL_KHR_robust_buffer_access_behavior
  - GL_KHR_robustness
  - GL_EXT_semaphore
  - GL_EXT_semaphore_fd
  - GL_NV_bgr
  - GL_NV_bindless_texture
  - GL_NV_blend_equation_advanced
  - GL_NV_conditional_render
  - GL_NV_copy_buffer
  - GL_NV_copy_image
  - GL_NV_draw_buffers
  - GL_NV_draw_instanced
  - GL_NV_draw_texture
  - GL_NV_draw_vulkan_image
  - GL_NV_EGL_stream_consumer_external
  - GL_NV_explicit_attrib_location
  - GL_NV_fbo_color_attachments
  - GL_NV_framebuffer_blit
  - GL_NV_framebuffer_multisample
  - GL_NV_generate_mipmap_sRGB
  - GL_NV_instanced_arrays
  - GL_NV_internalformat_sample_query
  - GL_NV_gpu_shader5
  - GL_NV_image_formats
  - GL_NV_occlusion_query_samples
  - GL_NV_non_square_matrices
  - GL_NV_pack_subimage
  - GL_NV_packed_float
  - GL_NV_packed_float_linear
  - GL_NV_path_rendering
  - GL_NV_pixel_buffer_object
  - GL_NV_polygon_mode
  - GL_NV_read_buffer
  - GL_NV_read_depth
  - GL_NV_read_depth_stencil
  - GL_NV_read_stencil
  - GL_NV_shader_noperspective_interpolation
  - GL_NV_shadow_samplers_array
  - GL_NV_shadow_samplers_cube
  - GL_NV_sRGB_formats
  - GL_NV_texture_array
  - GL_NV_texture_barrier
  - GL_NV_texture_border_clamp
  - GL_NV_texture_compression_latc
  - GL_NV_texture_compression_s3tc
  - GL_NV_texture_compression_s3tc_update
  - GL_NV_timer_query
  - GL_NV_viewport_array
  - GL_KHR_blend_equation_advanced
  - GL_OES_compressed_ETC1_RGB8_texture
  - GL_EXT_compressed_ETC1_RGB8_sub_texture
  - GL_OES_depth24
  - GL_OES_depth32
  - GL_OES_depth_texture
  - GL_OES_depth_texture_cube_map
  - GL_OES_copy_image
  - GL_OES_draw_buffers_indexed
  - GL_OES_draw_elements_base_vertex
  - GL_OES_texture_border_clamp
  - GL_OES_tessellation_point_size
  - GL_OES_tessellation_shader
  - GL_OES_texture_buffer
  - GL_OES_geometry_point_size
  - GL_OES_geometry_shader
  - GL_OES_gpu_shader5
  - GL_OES_shader_io_blocks
  - GL_OES_texture_view
  - GL_OES_primitive_bounding_box
  - GL_OES_EGL_image
  - GL_OES_EGL_image_external
  - GL_OES_EGL_image_external_essl3
  - GL_OES_EGL_sync
  - GL_OES_element_index_uint
  - GL_OES_fbo_render_mipmap
  - GL_OES_get_program_binary
  - GL_OES_mapbuffer
  - GL_OES_packed_depth_stencil
  - GL_OES_rgb8_rgba8
  - GL_OES_sample_shading
  - GL_OES_sample_variables
  - GL_OES_shader_image_atomic
  - GL_OES_shader_multisample_interpolation
  - GL_OES_standard_derivatives
  - GL_OES_surfaceless_context
  - GL_OES_texture_cube_map_array
  - GL_OES_texture_npot
  - GL_OES_texture_float
  - GL_OES_texture_float_linear
  - GL_OES_texture_half_float
  - GL_OES_texture_half_float_linear
  - GL_OES_texture_stencil8
  - GL_OES_texture_storage_multisample_2d_array
  - GL_OES_vertex_array_object
  - GL_OES_vertex_half_float
  - GL_OES_viewport_array
  - GL_ANDROID_extension_pack_es31a

Please describe your problem: No audio in any of applications including the settings app. The network seems to unable to connect. Clicking the music app crashes the whole Anbox (all windows will disappear) No sdcard and vold produce noises like "V vold : Waiting for FUSE to spin up..." The kernel message is flooded with lines like:

audit: type=1326 audit(1535281025.511:7266): auid=4294967295 uid=100000 gid=100000 ses=4294967295 pid=45881 comm="netd" exe="/system/bin/netd" sig=0 arch=c000003e syscall=93 compat=0 ip=0x7f6eb16b8ac7 code=0x7ffc0000

Is this issue related to the AppArmor or premission leakage? Following log may related to the sdcard problem:

08-26 10:42:00.678   190   190 E sdcard  : Error setting RLIMIT_NOFILE, errno = 1
08-26 10:42:00.678   190   190 W sdcard  : Device explicitly disabled sdcardfs
08-26 10:42:00.679   190   190 E sdcard  : failed to mount fuse filesystem: Operation not permitted
08-26 10:42:00.679   190   190 E sdcard  : failed to fuse_setup

What were you expecting?: Working Android functions. Settings to the DNS server, IP address, IPv6 support, Proxy.

Additional info:

morphis commented 6 years ago

The problem seems to boil down to missing support in your kernel for FUSE mounts in unprivileged user namespaces. The Ubuntu kernel comes with support for this for quite a while but I am not sure what the state of this is in the Debian kernel you're using. @zhsj Any idea?

The audit messages in the kernel log are normal and not a problem as the snap is running in devmode which enables seccomp and AppArmor only in complain mode.

Regarding Settings to the DNS server, IP address, IPv6 support, Proxy. I am not sure what you mean as you primarily describe the problem with the failing sdcard process. To separate things can you file a separate bug report for the problem you're seeing for any missing or not working network function?

zhsj commented 6 years ago

@morphis How can I check whether the fuse module supports unprivileged user namespace? Is there a kernel config? Is this feature in upstream? I don't think Debian has magic patch to upstream fuse module.

$ grep -i fuse /boot/config-4.17.0-3-amd64 
CONFIG_FUSE_FS=m
kde99 commented 6 years ago

I'm also having the same FUSE spamming problem problem with a freshly installed anbox on gentoo I have CONFIG_FUSE_FS=m too. 

version: 4
snap-revision: 144
cpu:
  arch:  x86
  brand:        Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz
  features: 
    - aes
    - sse4_1
    - sse4_2
    - avx
os:
  name: Gentoo
  version: 
  snap-based: true
kernel:
  version: Linux version 4.14.65-gentoo (root@pen) (gcc version 8.2.0 (Gentoo 8.2.0-r2 p1.2)) #4 SMP Mon Sep 3 12:20:52 CEST 2018
  binder: true
  ashmem: true
graphics:
  egl:
    vendor: Mesa Project
    version: 1.4 (DRI2)
    extensions:
      - EGL_ANDROID_native_fence_sync
      - EGL_CHROMIUM_sync_control
      - EGL_EXT_buffer_age
      - EGL_EXT_create_context_robustness
      - EGL_EXT_image_dma_buf_import
      - EGL_EXT_image_dma_buf_import_modifiers
      - EGL_KHR_config_attribs
      - EGL_KHR_create_context
      - EGL_KHR_create_context_no_error
      - EGL_KHR_fence_sync
      - EGL_KHR_get_all_proc_addresses
      - EGL_KHR_gl_colorspace
      - EGL_KHR_gl_renderbuffer_image
      - EGL_KHR_gl_texture_2D_image
      - EGL_KHR_gl_texture_3D_image
      - EGL_KHR_gl_texture_cubemap_image
      - EGL_KHR_image
      - EGL_KHR_image_base
      - EGL_KHR_image_pixmap
      - EGL_KHR_no_config_context
      - EGL_KHR_reusable_sync
      - EGL_KHR_surfaceless_context
      - EGL_EXT_pixel_format_float
      - EGL_KHR_wait_sync
      - EGL_MESA_configless_context
      - EGL_MESA_drm_image
      - EGL_MESA_image_dma_buf_export
      - EGL_NOK_texture_from_pixmap
      - EGL_WL_bind_wayland_display
  gles2:
    vendor: Intel Open Source Technology Center
    vendor: OpenGL ES 3.0 Mesa 18.0.5
    extensions:
      - GL_ANGLE_texture_compression_dxt3
      - GL_ANGLE_texture_compression_dxt5
      - GL_APPLE_texture_max_level
      - GL_EXT_blend_func_extended
      - GL_EXT_blend_minmax
      - GL_EXT_clip_cull_distance
      - GL_EXT_color_buffer_float
      - GL_EXT_compressed_ETC1_RGB8_sub_texture
      - GL_EXT_discard_framebuffer
      - GL_EXT_disjoint_timer_query
      - GL_EXT_draw_buffers
      - GL_EXT_draw_buffers_indexed
      - GL_EXT_draw_elements_base_vertex
      - GL_EXT_frag_depth
      - GL_EXT_map_buffer_range
      - GL_EXT_multi_draw_arrays
      - GL_EXT_occlusion_query_boolean
      - GL_EXT_polygon_offset_clamp
      - GL_EXT_read_format_bgra
      - GL_EXT_robustness
      - GL_EXT_separate_shader_objects
      - GL_EXT_shader_integer_mix
      - GL_EXT_texture_border_clamp
      - GL_EXT_texture_compression_dxt1
      - GL_EXT_texture_filter_anisotropic
      - GL_EXT_texture_format_BGRA8888
      - GL_EXT_texture_rg
      - GL_EXT_texture_sRGB_decode
      - GL_EXT_texture_type_2_10_10_10_REV
      - GL_EXT_unpack_subimage
      - GL_INTEL_performance_query
      - GL_KHR_blend_equation_advanced
      - GL_KHR_context_flush_control
      - GL_KHR_debug
      - GL_KHR_no_error
      - GL_KHR_robustness
      - GL_MESA_shader_integer_functions
      - GL_NV_draw_buffers
      - GL_NV_fbo_color_attachments
      - GL_NV_read_buffer
      - GL_NV_read_depth
      - GL_NV_read_depth_stencil
      - GL_NV_read_stencil
      - GL_OES_EGL_image
      - GL_OES_EGL_image_external
      - GL_OES_EGL_sync
      - GL_OES_compressed_ETC1_RGB8_texture
      - GL_OES_depth24
      - GL_OES_depth_texture
      - GL_OES_depth_texture_cube_map
      - GL_OES_draw_buffers_indexed
      - GL_OES_draw_elements_base_vertex
      - GL_OES_element_index_uint
      - GL_OES_fbo_render_mipmap
      - GL_OES_get_program_binary
      - GL_OES_mapbuffer
      - GL_OES_packed_depth_stencil
      - GL_OES_required_internalformat
      - GL_OES_rgb8_rgba8
      - GL_OES_sample_shading
      - GL_OES_sample_variables
      - GL_OES_shader_multisample_interpolation
      - GL_OES_standard_derivatives
      - GL_OES_stencil8
      - GL_OES_surfaceless_context
      - GL_OES_texture_3D
      - GL_OES_texture_border_clamp
      - GL_OES_texture_float
      - GL_OES_texture_float_linear
      - GL_OES_texture_half_float
      - GL_OES_texture_half_float_linear
      - GL_OES_texture_npot
      - GL_OES_vertex_array_object
      - GL_OES_vertex_half_float

If anyone has ideas what other kernel parameter can be missing I'd be happy to add it to my kernel and test :D

comword commented 6 years ago

After some research FUSE mounts in unprivileged containers, this issue is related to the kernel and lxc continainer. Please consider giving an option to use the privileged continainer rather than modify the kernel.

zhsj commented 6 years ago

There's an option --privileged to anbox container-manager.

morphis commented 6 years ago

If you want to run a privileged container just set snap set anbox container.privileged=true. However a privileged container is not the right answer as it compromises security and effectively it becomes easy for Android apps to own your entire system as we don't have much other security protection in place (we can't use SELinux like regular Android builds do). Also running Anbox in privileged mode will break in other places as the Android rootfs has a shifted user+group ids to the new base id 100000.

The right answer to the FUSE problem is that run the sdcard daemon process from the Android system outside of the user namespace. That removes the unprivileged mount problem but brings in a new problem as we need to tell sdcard to properly translate all ids to our base one at 100000.

Your kernel should support FUSE in user namespaces when /sys/module/fuse/parameters/userns_mounts is present.

kde99 commented 6 years ago

So the solutions would be either to patch or kernels with that Ubuntu patch or somehow hope that running in privileged mode won't break it or to have FUSE work from inside out? Wouldn't it be easier to like try maybe NBD to mount the sdcard or maybe NFS apart from the performance penalty it would bring?

morphis commented 6 years ago

NBD or NFS wont help as the sdcard FUSE file system serves a very Android specific need. Have a read at http://androidxref.com/6.0.0_r5/xref/system/core/sdcard/sdcard.c#47 and https://www.xda-developers.com/diving-into-sdcardfs-how-googles-fuse-replacement-will-reduce-io-overhead/ for more details. Working around the sdcard FUSE file system is not an option, sadly.

kde99 commented 6 years ago

Well that's sad then. Is there any way I could get that one patch I need for FUSE?

zhsj commented 6 years ago

FWIW, fuse in user namespace is accepted in upstream tree since 4.18.

https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.18-FUSE

morphis commented 6 years ago

@kde99 I can't say if its a single patch or multiple. The link @zhsj provided should be a good starting point for further investigations.

kde99 commented 6 years ago

@zhsj I updated to 4.18.5 and "ls: cannot access '/sys/module/fuse/parameters/userns_mounts': No such file or directory" also now anbox is now failing to even start up with "SIGBUS" in art paste of log . Back with the older kernel it starts well though.

EDIT: It starts under 4.14.65 well but FUSE fails because of the missing stuff.

morphis commented 6 years ago

@kde99 The SIGBUS crash on kernels >= 4.18 was just fixed with https://github.com/anbox/anbox-modules/pull/9 A newer anbox-modules package is already available in the anbox-support PPA (https://launchpad.net/~morphis/+archive/ubuntu/anbox-support/). If you manually installed the kernel modules or use packages for a different distribution they need to be updated to include the changes from the mentioned PR.

/sys/module/fuse/parameters/userns_mounts does not need to exist on plain upstream kernels as I suspect the upstream and Ubuntu implementations are slightly different. The check in the container-manager startup script just prints a warning but still continues to start the manager.

kde99 commented 6 years ago

@morphis Ok so after updating the kernel modules It does work under 4.18.5 well FUSE mounted too. I can see the SD from Total commander but suffering from #443 still but that's not as bad as not being able to start. Thanks for helping ^^