Make the publisher buildable as a standalone project

[adam-stanek]

Similar to issue #111 we should do the same thing for publisher repo.

Here is a snippet of possible Dockerfile:

FROM golang:alpine AS build

RUN apk add --update bash curl git && \
    curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh

WORKDIR /go/src/github.com/analogj/lodestone-publisher/

ADD Gopkg.toml ./
ADD Gopkg.lock ./
ADD cmd ./cmd
ADD pkg ./pkg

RUN dep ensure
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o lodestone-fs-publisher ./cmd/fs-publisher/fs-publisher.go

FROM alpine
COPY --from=build /go/src/github.com/analogj/lodestone-publisher/lodestone-fs-publisher /lodestone-fs-publisher
CMD ["/lodestone-fs-publisher"]

[dskaggs]

@adam-stanek Couple of questions for this one:

1. Do you have a sense of what it might take to move the publisher repo over to using modules?
2. Since this has both fs-publisher and email-publisher, would we want to follow the same pattern as we just set up with the processor and have multiple Docker files and multiple DockerHub repos?

[dskaggs]

After playing with this for a bit using your Dockerfile suggestion above, I think we're going to have to combine this repo and the lodestone-storage repo. The lodestone-fs-publisher is launched as a service inside the https://github.com/LodestoneHQ/lodestone-storage repo (see https://github.com/LodestoneHQ/lodestone-storage/blob/fa67ffb379c418590b47b747a3a22b310411e9f2/rootfs/etc/services.d/fs-publisher/run#L7 ) and has a dependency on Minio.

[adam-stanek]

Do you have a sense of what it might take to move the publisher repo over to using modules?

I think it should be pretty straightforward. But I would have to take a closer look if all the project dependencies are usable as go modules. They most likely are though.

 Since this has both fs-publisher and email-publisher, would we want to follow the same pattern as we just set up with the processor and have multiple Docker files and multiple DockerHub repos?

Yes, I think it would be the best solution. I want to have a look at the email-publisher, I might send some PR your way after I get it running.

[adam-stanek]

After playing with this for a bit using your Dockerfile suggestion above, I think we're going to have to combine this repo and the lodestone-storage repo. The lodestone-fs-publisher is launched as a service inside the https://github.com/LodestoneHQ/lodestone-storage repo (see https://github.com/LodestoneHQ/lodestone-storage/blob/fa67ffb379c418590b47b747a3a22b310411e9f2/rootfs/etc/services.d/fs-publisher/run#L7 ) and has a dependency on Minio.

Yes, I noticed that myself as well. I had to actually built it separately because of that because I am running custom minio setup with hardened policies and backup snapshot repository for Elasticsearch.

To be honest I am not 100% why we even have fs-publisher. It notifies the stack on new files, but I get the feeling it should be possible to do the same thing with only a Minio (it supports AMQP as well). In that case we could focus solely on email-publisher and simplify the stack a bit.

[dskaggs]

I think it came about before Minio was part of the stack. I personally would love the option to not have to use Minio and just point it to the directory where my documents already live. Seems like a duplication of file storage to me.

[adam-stanek]

It would make sense. But I am not sure it is worth developing custom solution. Minio is already battle tested + it has many additional features. Also it can easily be switched to hosted S3 service and leverage for example Amazon Glacier for long term storage without having to worry about backups.

[dskaggs]

Fair enough. It's not like we have a lot of extra engineering bandwidth to be able to afford to go looking for extra work :)

[adam-stanek]

Do you have a sense of what it might take to move the publisher repo over to using modules?

So I had a closer look at the publisher repo and it won't be that easy. There is 1 forked library with unreleased version which isn't packaged properly. Sadly the required fork didn't find its way to the upstream because it requires linux only changes to otherwise cross-platform library. We can create our own fork, but honestly I would rather look for different solution to the problem - ie. consider dropping the fs-publisher completely.