search

artemkin / git-lfs-server

Git LFS server
Other
140 stars 20 forks source link

Git LFS Server Fails to Create Files While Running Inside Docker Container With Volume as Objects Directory #16

Closed pewdeveloper closed 4 years ago

pewdeveloper commented 4 years ago

This issue might not be caused by git-lfs-server but wanted to see if anyone else was struggling with similar problems getting git-lfs-server deployed inside a docker container with a mounted data volume as the objects directory.

This is using git-lfs-server release number 0.3.1

When deploying git-lfs-server in a docker container everything works mostly as expected. However if you try to use a volume to persist data in between container instantiations suddenly this error starts to appear in the logs:

git-lfs-server_1  | 2019-10-22 18:59:18.581475Z Error 172.20.0.1 Failure: with_file_atomic could not create file

If I check the volume directory it appears that the git-lfs-server can create empty directories but not files. If I log into the container I seem to have all the required permissions to read and write files on the volume mount and can create large test files using dd

After letting the server rest idle for a few minutes this error shows up in the logs:

git-lfs-server_1  | 2019-10-22 19:51:08.225611Z Error 172.20.0.1 Exception: ("Writer error from inner_monitor" "writer fd unexpectedly closed "
git-lfs-server_1  |   (writer
git-lfs-server_1  |     ((id 413)
git-lfs-server_1  |       (fd
git-lfs-server_1  |         ((file_descr 20)
git-lfs-server_1  |           (info
git-lfs-server_1  |             (socket
git-lfs-server_1  |               ((listening_on
git-lfs-server_1  |                  ((type_ ((family PF_INET) (socket_type SOCK_STREAM)))
git-lfs-server_1  |                    (fd
git-lfs-server_1  |                      ((file_descr 8)
git-lfs-server_1  |                        (info
git-lfs-server_1  |                          (replaced
git-lfs-server_1  |                            (listening
git-lfs-server_1  |                              (previously_was
git-lfs-server_1  |                                (replaced
git-lfs-server_1  |                                  ((socket (bound_on 0.0.0.0:55600))
git-lfs-server_1  |                                    (previously_was
git-lfs-server_1  |                                      (socket
git-lfs-server_1  |                                        ((family PF_INET)
git-lfs-server_1  |                                          (socket_type SOCK_STREAM))))))))))
git-lfs-server_1  |                        (kind (Socket Passive)) (supports_nonblock true)
git-lfs-server_1  |                        (have_set_nonblock true) (state (Open Empty))
git-lfs-server_1  |                        (watching
git-lfs-server_1  |                          ((read (Watch_once Empty)) (write Not_watching)))
git-lfs-server_1  |                        (watching_has_changed false) (num_active_syscalls 1)
git-lfs-server_1  |                        (close_finished Empty)))))
git-lfs-server_1  |                 (client 172.20.0.1:35346))))
git-lfs-server_1  |           (kind (Socket Active)) (supports_nonblock true)
git-lfs-server_1  |           (have_set_nonblock true) (state Closed)
git-lfs-server_1  |           (watching ((read Not_watching) (write Not_watching)))
git-lfs-server_1  |           (watching_has_changed false) (num_active_syscalls 0)
git-lfs-server_1  |           (close_finished (Full ()))))
git-lfs-server_1  |       (monitor
git-lfs-server_1  |         (((name (id 14609)) (here ()) (id 14609) (has_seen_error false)
git-lfs-server_1  |            (is_detached true))
git-lfs-server_1  |           ((name "") (here ()) (id 2) (has_seen_error false)
git-lfs-server_1  |             (is_detached true))))
git-lfs-server_1  |       (inner_monitor
git-lfs-server_1  |         (((name (id 14610)) (here ()) (id 14610) (has_seen_error true)
git-lfs-server_1  |            (is_detached true))
git-lfs-server_1  |           ((name "") (here ()) (id 2) (has_seen_error false)
git-lfs-server_1  |             (is_detached true))))
git-lfs-server_1  |       (background_writer_state Stopped_permanently)
git-lfs-server_1  |       (background_writer_stopped (Full ())) (syscall Per_cycle)
git-lfs-server_1  |       (bytes_received 225) (bytes_written 0) (scheduled_bytes 0)
git-lfs-server_1  |       (scheduled_back 0) (back 0) (close_state Open) (close_finished Empty)
git-lfs-server_1  |       (close_started Empty) (num_producers_to_flush_at_close 0)
git-lfs-server_1  |       (flush_at_shutdown_elt (<opaque>))
git-lfs-server_1  |       (check_buffer_age
git-lfs-server_1  |         (((writer <opaque>) (maximum_age 2m)
git-lfs-server_1  |            (bytes_received_at_now_minus_maximum_age 0)
git-lfs-server_1  |            (bytes_received_queue ()) (times_received_queue ()) (bytes_seen 0)
git-lfs-server_1  |            (too_old Empty))))
git-lfs-server_1  |       (consumer_left Empty) (raise_when_consumer_leaves false)
git-lfs-server_1  |       (open_flags (Full (Ok (rdwr)))) (line_ending Unix))))

Here is my git/lfs version information:

git-lfs/2.3.4 (GitHub; linux amd64; go 1.8.3)
git version 2.17.1
pewdeveloper commented 4 years ago

Here is my Dockerfile and docker-compose.yml if anyone wants to try to reproduce:

Dockerfile

FROM bitnami/minideb:stretch

WORKDIR /usr/src/app

RUN install_packages curl ca-certificates

RUN curl -LJ -o ./lfs_server.tar.gz https://github.com/artemkin/git-lfs-server/releases/download/v0.3.1/lfs_server-0.3.1-linux.x64.tar.gz

RUN tar xvf ./lfs_server.tar.gz && mv ./lfs_server/* . && rm -rf ./lfs_server ./lfs_server.tar.gz

RUN apt-get --purge -y remove curl ca-certificates

EXPOSE 55600/tcp
VOLUME /usr/src/app/.lfs/objects

RUN useradd -MUs /bin/bash -d /usr/src/app lfs
RUN chown -R lfs:lfs /usr/src/app
USER lfs

CMD ["./lfs_server.sh", "-p 55600", "-s 0.0.0.0", "-verbose"]

docker-compose.yml

version: "3.3"                                                                                                                                                                                                                                  services: 
    git-lfs-server:
        build: .
        image: lfs_server
        ports: 
          - "55600:55600"
        volumes: 
          - <source_directory_for_mounted_volume>:/usr/src/app/.lfs/objects:Z
pewdeveloper commented 4 years ago

I was able to resolve this myself.

I tracked down the issue to the call Unix.rename

For some reason this call fails if the temp dir is outside of your mounted volume. I changed my configuration so the whole .lfs directory is inside the mounted volume and operation returned to normal. Closing now, I hope this can help others in the future.