Migration process from GridFS to homedirs assetstore

[craig-willis]

With v0.2 we need the ability to migrated all user data from the GridFS assetstore to the new homedirs assetstore. This should be as simple as using the file/{id}/move API.

Completion criteria:

• [x] Script to migrate data between assetstores
• [x] Migrated data on staging instance
• [x] Verify

[craig-willis]

Ran a quick test on staging instance to move from GridFS to homedirs asset store and am seeing the following error:

Traceback (most recent call last):
  File "/girder/girder/api/rest.py", line 620, in endpointDecorator
    val = fun(self, args, kwargs)
  File "/girder/girder/api/rest.py", line 1208, in PUT
    return self.handleRoute('PUT', path, params)
  File "/girder/girder/api/rest.py", line 947, in handleRoute
    val = handler(**kwargs)
  File "/girder/girder/api/access.py", line 42, in wrapped
    return fun(*args, **kwargs)
  File "/girder/girder/api/rest.py", line 467, in wrapped
    val = fun(*args, **kwargs)
  File "/girder/girder/api/describe.py", line 702, in wrapped
    return fun(*args, **kwargs)
  File "/girder/girder/api/v1/file.py", line 415, in moveFileToAssetstore
    file=file, user=user, assetstore=assetstore, progress=ctx)
  File "/girder/girder/models/upload.py", line 425, in moveFileToAssetstore
    file=file, user=user, size=int(file['size']), assetstore=assetstore)
  File "/girder/girder/models/upload.py", line 304, in createUploadToFile
    assetstore = self.getTargetAssetstore('file', file, assetstore)
  File "/girder/girder/models/upload.py", line 275, in getTargetAssetstore
    event = events.trigger('model.upload.assetstore', eventParams)
  File "/girder/girder/events.py", line 310, in trigger
    handler['handler'](e)
  File "/girder/plugins/wt_home_dir/server/__init__.py", line 72, in handler
    path = pathlib.Path(h.getPath(event))
  File "/girder/plugins/wt_home_dir/server/lib/EventHandlers.py", line 18, in getPath
    force=True)
  File "/girder/girder/utility/path.py", line 248, in getResourcePath
    parentModel = doc['parentCollection']
KeyError: 'parentCollection'

The event handler in the wt_home_dirs plugin (https://github.com/whole-tale/wt_home_dirs/blob/stable/server/lib/EventHandlers.py#L17) is calling the Girder path utility getResourcePath method (https://github.com/girder/girder/blob/master/girder/utility/path.py#L223). It's not clear to me why the type is folder in this case.

[Xarthisius]

It's caused by AssetstoreQueryHandler.getResourceType returning "folder" unconditionally. Moving file triggers an event that carries file object.

[craig-willis]

@Xarthisius I've converted my migration script to use the internal API (via girder-shell) and hit a snag with the webdav mount. I need an environment with both girder-shell and girderfs support. I've tried installing the fuse packages and mount.c from gwvolman into an image derived from wholetale/girder:stable.

FROM wholetale/girder:stable

RUN apt-get update -qqy && \
  DEBIAN_FRONTEND=noninteractive apt-get -qy install \
    build-essential \
    vim \
    git \
    kmod \
    wget \
    python \
    fuse \
    davfs2 \
    libffi-dev \
    libssl-dev \
    libjpeg-dev \
    zlib1g-dev \
    libfuse-dev \
    libpython-dev && \
  apt-get -qqy clean all && \
  echo "user_allow_other" >> /etc/fuse.conf && \
  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN wget https://bootstrap.pypa.io/get-pip.py && python get-pip.py

RUN pip install bson git+https://github.com/whole-tale/girderfs@v0.2#egg=girderfs

ADD https://raw.githubusercontent.com/whole-tale/gwvolman/master/mount.c /tmp/mount.c
RUN gcc -Wall -fPIC -shared -o /usr/local/lib/container_mount.so /tmp/mount.c -ldl -D_FILE_OFFSET_BITS=64 && \
   rm  /tmp/mount.c && \
   chmod +x /usr/local/lib/container_mount.so && \
   echo "/usr/local/lib/container_mount.so" > /etc/ld.so.preload

I'm running with the same docker arguments as gwvolman:

 docker run \
     -it \
     --name wholetale_migrate \
     --label traefik.enable=false \
     -v /:/host \
     -v /var/cache/davfs2:/var/cache/davfs2 \
     -v /run/mount.davfs:/run/mount.davfs \
     --device /dev/fuse \
     --cap-add SYS_ADMIN \
     --cap-add SYS_PTRACE \
     --network wt_mongo \
     --entrypoint /bin/bash \
     craigwillis/wholetale-migrate

However, the mount itself is failing, likely due to problems with my fuse install process:

girderfs -c wt_home --api-url https://girder.stage.wholetale.org/api/v1 --api-key key /tmp/migrate/user folderId
mount.davfs: can't mount using fuse kernel file system
mount.davfs: trying coda kernel file system
mount.davfs: no free coda device to mount
...

Is there something obvious I've missed?

[Xarthisius]

You're missing proper user configuration, add this to your image:

RUN userdel node && useradd -g 100 -G 100 -u 1000 -s /bin/bash wtuser

The question is: do you really want to mount outside of wholetale-migrate container? I thought it's not necessary. If that's the case you can just drop the mount.c bit or simply rm /etc/ld.so.preload in your existing container.

[craig-willis]

Thanks, @Xarthisius. You're right, I don't need to mount outside the container -- I didn't think that through.

[craig-willis]

OK, preliminary migration script and Dockerfile have been pushed to:

https://github.com/craig-willis/wholetale-migrate/

The README walks through the complete process. I've run through the migration on staging already.

At this point, I think it's working except for zero byte files.

Migration script output available in gist:

https://gist.github.com/craig-willis/62dca118842235f840ef10b958d3cd51

[Xarthisius]

Migrated and deployed in production