search

dyvenia / viadot

A simple library to move data between places.
https://viadot.docs.dyvenia.com
MIT License
25 stars 40 forks source link

viadot compability with MacOS #848

Closed k-wolski closed 9 months ago

k-wolski commented 9 months ago

As Velux is starting migration to MacOS, we need to test if everything with viadot will work on new M chips they are using.

Image: https://github.com/dyvenia/viadot/pkgs/container/viadot%2Fviadot/172584182?tag=0.4.25 docker-compose: https://github.com/dyvenia/viadot/blob/main/docker/docker-compose.yml

trymzet commented 9 months ago

@kwolskiDyvenia might need to re-add ARM build of the image, see https://github.com/dyvenia/viadot/pull/620. @Rafalz13 or his team should be able to help with that

k-wolski commented 9 months ago

Building arm image of viadot will be handled in separate issue.

Regarding tests on MacOS, everything is working fine. Important thing is to have MacOS updated to at least 12.4, but ideally to latest available version (14.3 currently). Earlier versions are crashing during start of docker/podman.

Docker desktop: It just works, so it's enough to install docker desktop + docker-compose and run scripts like on the Windows. It's not that useful though as client is using Podman because of license.

Podman Instruction prepared by @trymzet for podman + Windows is mostly correct on MacOS. Few changes that are needed:

  1. --userns=keep-id needs to be added to map user ID from host to viadot user inside of container. Then all of the files can be modified inside of container as viadot:viadot, while on host it will be user:user, eg. kwolski:kwolski
  2. .bashrc needs to be created manually, as on MacOS zsh is default shell
  3. For -v diffferent path needs to be added as /home/$USER is not valid for MacOS, while /Users/$USER is.

onboarding_1.docx

To summarize it - we are ok with this image, and I didn't face issues we had ~1 year ago.

k-wolski commented 9 months ago

When I tested setup on a clean system, I learnt that it's not bulletproof solution using --userns=keep-id. To avoid issues for clients using MacOS, instruction was slightly changed, and all changes can be summarized like this:

  1. -u 0:0 should be added during run of container. Because podman is running on user account, not root, podman will map IDs in such way, that userId on laptop will be mapped to root in container. No changes will be needed to any folder, and new files/folder created in container will be owned by laptop userID.
  2. --allow-root is necessary during startup of jupyter_lab, as inside container we are treated as root

Updated version: onboarding__1.docx