This project is bigger than me

[danielquinn]

...and I don't have a whole lot of time to donate to it.

Since I started it back in 2015, Paperless has grown far beyond what I ever expected it to be. With 5000+ stars on Github and a surprising popularity in Germany, issues are posted or commented on on an almost daily basis.

With the little time that I have (baby on the way!), I try to roll out fixes & features, but every change tends to add instabilities on platforms I don't personally use. There are people using this project on bare-metal Linux installs, Docker installs on Synology boxes, cloud installations, Rasberry Pis, and FreeBSD machines -- and that's only the ones I know about from issues users have posted.

If this project is going to survive & grow, there has to be more people driving it. Primarily, we need people who the rest of us can trust, but also people willing to do the boring work of adding tests and testing frameworks for the various platforms.

I'd like to transition Paperless to more of a collective effort, either as part of the @Jazzband, or some other means -- I'm open to suggestions. I'm happy to stay on to audit pull requests or make suggestions, but it's becoming clear that Paperless needs more from me than I'm able to give it, and so I'm thinking we need to expand the pool.

Here's what I'm suggesting: I believe GitHub can be configured to require that merges be reviewed by one or more other collaborators before code is merged. If I can figure out how to do this (maybe I need to create an organisation? Advice on this is appreciated.) then we could have a system where we have more people who can create & merge more stuff, but where merges require review from at least one other person -- that'll keep people honest.

If GitHub can't do this, then I'm also happy to move to GitLab unless there are strong objections.

IMHO, the first mission of this collective should be to get us to 100% test coverage and to update Travis (or something else?) to execute those tests in different environments (if possible). After that, I imagine some of you have your own priorities regarding where this should go. I'm excited to see what you have in mind.

I'm tagging a few people specifically who have been more heavily involved with Paperless than others just to give them a heads-up. If any of you lot are interested in taking a more primary role in this project, please comment below. Maybe we can get an idea of what kinds of resources/interest is out there. This is not an exclusive list, so if you don't appear here but feel you've contributed significantly (or just want to) it's because I didn't recognise your username from the autocomplete:

:+1: @Strubbl @philippeowagner @jat255 @ahyear @avichalp @BastianPoe @CkuT @ddddavidmartin @erikarvstedt @MasterofJOKers @pitkley @ovv

If no one steps up, nothing will change. That is to say, issues will continue to pop up and people from the community will (hopefully) try to help people with questions. I will try to keep on top of pull requests and chat in the issue queue when I have time, but as we've already seen, I don't have a lot of time.

[ahyear]

Hello @danielquinn ,

I'm happy to participate, unfortunatly i am not a good dev ( not a dev at all!). I can help on testing et beta testing the docker part but i won't be on any help en correcting bugs or managing the repo...

I certainly hope that this project will grow to be a new standard as i have not seen anything as efficient as this one.

[philippeowagner]

@danielquinn congratulations! and thank you very much for all the efforts and time you invested in paperless. Maintaining open source projects is hard, especially in the long run.

Unfortunately I can hardly keep up with my own open source projects like django-hijack, django-background-tasks, etc.

Even if I would like to, it will not be possible for me to participate as a lead in this project at the moment.

Thanks again Daniel for all the work on and around paperless.

[ovv]

Thanks for everything @danielquinn. Sadly I have to resonate with the others I don't have time (for the moment) to follow the project as much as I want.

[danielquinn]

Thanks for responding guys. @philippeowagner & @ovv: I know exactly where you're coming from. I'm trying to keep the bulk of my Free software contributions focused on Aletheia, and with the kid due in a few days, I figure that even that sounds optimistic. Your contributions thus far have been greatly appreciated :-)

@ahyear it's cool that you can't do dev stuff, but testing is super important too! How do you feel about me tagging you on the occasional pull request to ask for a review? That is to say, you'd have to pull the changes and run them to check that things are working as expected?

[CkuT]

Congratulations @danielquinn ! Thanks for this great project ;)

I really want to improve paperless, and I have some ideas but don't have many time at the moment. I think I'll have more time available in few months to work on paperless. However, I would be glad to be tagged for reviews in the meantime.

[JensPfeifle]

Thanks as well from a recent convert from MayanEDMS. I’ve been looking for a project I can contribute to and would like to help. As a student I’m not very experienced but may have a little more time than others ;)

I was just reading through some of the open issues, and think that quite a few of them could be closed right away, e.g. #241, #292, #327, or #446. Removing some of the “clutter“ in the issue list and making it more clear where contributions are needed may help in getting more people onboard.

[syntonym]

Configuring github so that any pull requests needs X reviews should be straightforward and should not need a new organization. The documentation mentions a setting in the protected branch settings where the behavior can be configured. Users with write access or admin access to the repository count as reviewers.

Congratulations on the baby!

[ahyear]

@danielquinn you can add me, I will try my best to help the project. But even if you slow down your involvement, you will still be there right?

By the way I just had my second baby as well 9 days ago ;)

So I understand your situation....

[bmsleight]

testing frameworks for the various platforms

bare-metal Linux installs, Docker installs on Synology boxes, cloud installations, Rasberry Pis, and FreeBSD machines

I would like to give this a crack.

Enjoy the baby! Although I think it just a way of generating a Birth Certificate, so your can store it in paperless ;)

[danielquinn]

Thanks for all of your advice everyone, especially to @syntonym who pointed me in the right direction regarding branch protection and merge rules.

I've tweaked this project to require a minimum of 2 :+1: reviews from "collaborators" before a merge to master is permitted. However, I don't have any collaborators on the project yet, so the behaviour remains the same (as an admin, I can merge whatever I want).

So, who here wants to be a collaborator? This would mean that you could be tagged for a request to review a pull request and that you could be one of the two required reviewers to merge something.

As for @ahyear's question: yes, I will still be around. I'm just short on time/resources and I don't want this project to die just because I'm busy. This is basically an attempt to improve the bus factor of Paperless, not a means of abdicating my role in it.

The fixing of the bus factor starts with other people volunteering to be collaborators/reviewers, and until then, this project will move forward... slowly, but it'll still be moving.

[jat255]

@danielquinn I had thus far failed to pipe up, but I'd be happy to be a collaborator/reviewer. I've been using the project regularly for the past few months, and I have a few ideas on how to improve some things (if I can find the time!)

[MasterofJOKers]

I'm happy to help with reviewing stuff whenever time permits.

[ahyear]

@danielquinn asI said, I will be happy to help to. (as much as i can...)

[kevin2107]

@danielquinn I'd love to start helping, I've recently made the decision to go paperless and would like to begin contributing bug fixes and features

[ovv]

I was thinking about this and #479 . I believe it could be a good idea to create a paperless organization that way some non-essential part could be taken out of the primary repository and maintain by other people more easily. It will also reduce the load on @danielquinn.

By non-essential part I'm thinking about documentation, docker setup, other installations method like the script in #479 if we decide to expand on it or the ansible role I maintain.

[JensPfeifle]

@danielquinn, If you can give me permission to assign labels I would start by sorting through the issue queue.

[danielquinn]

Alright it sounds like I'll have to convert this project into an organisation. I'll try to do that this weekend and will keep this issue updated with whatever I figure out.

If anyone here has some tips regarding how to do this, by all means, share it :-)

[philippeowagner]

Create a new organisation https://github.com/organizations/new, eg. paperless. Transfer the paperless repo to paperless. That's it.

See https://help.github.com/articles/transferring-a-repository/ for more info.

[danielquinn]

Ok I've now done the following:

• Created a new Paperless organisation. It turns out paperless was taken, as was paperless-project, and the-paperless (pretty crowded space I guess), so I ended up with the-paperless-project :-)
• I added a bunch of the people who spoke up here as members: @ovv, @ahyear, @MasterofJOKers, @jat255, @bmsleight, and @CkuT. As for @JensPfeifle & @kevin2107, I appreciate your enthusiasm, but as you haven't submitted any pull requests yet, I'm not ready to add you as a member. Feel free to find an issue you wanna help out with, send a PR and once your contributions come through, we can talk about granting write permissions.

If I've done this right, each one of the organisation members will have write permissions to the project to create branches or even additional projects, but as the paperless/master branch is protected, requiring 2+ approvals to merge, a member shouldn't be able to arbitrarily merge stuff into master. Please let me know if that's correct.

Additionally, I know that @JensPfeifle offered to go around and work on labelling stuff in the issue queue. That's awesome and really appreciated, but I can't figure out how to do that without giving you write access to the code, and I'm not sure how I feel about offering that for someone who hasn't submitted a PR yet. Thoughts from the rest of you on this are appreciated.

Let me know if anyone thinks I've screwed this up, eh?

[bmsleight]

Cool [Paperless organisation]https://github.com/the-paperless-project)!

a member shouldn't be able to arbitrarily merge stuff into master

Most of the open pull requests are for docker, guess I have to start learning docker!

[pitkley]

Hey @danielquinn, sorry for not responding sooner. I'm glad you reached out to the great community of users and contributors of Paperless in aiding you and the project itself. I think that adopting additional maintainers is a great step in the right direction, and requiring 2+ approvals is the right way to still ensure the quality of Paperless, even if you personally can't check or "vouch" for every change introduced.

I have seen that you already have added quite a few members to the organization, which I find awesome! :+1: Should you think that one more helps, feel free to add me. In any case I'll continue contributing to this project as I have in the past (and as time allows), either if I stumble upon something where I feel like I can support, or if I'm specifically tagged on something where the community thinks my input could help!

Also a big thank you to everyone in this thread, without the support and contributions extended by all of you Paperless wouldn't be as great of a tool! :tada:

[danielquinn]

I was hoping you'd be interested @pitkley! I'll send your invite now :-)

[ovv]

If it's ok with everyone, and after a bit of cleanup, I plan to move ovv/ansible-role-paperless to the organization.

[philippeowagner]

@ovv I personally think it's a good idea to find all loose parts in one place. But we should consider have the licences aligned. I noticed ansible-role-paperless is licensed unter the MIT License and paperless itself under the GNU General Public License v3.0.

[bmsleight]

HI Any chance of a review of my PRs ? @ovv @ahyear @MasterofJOKers @jat255 @CkuT @pitkley ?

499 #500 #501 #502

[pitkley]

@danielquinn apparently the move to the organization broke Travis CI, at least it doesn't seem like any of the PRs opened since then are building. I have requested approval from you for Travis CI to be able to access the organization.

I'm not 100% sure if that was the right thing, maybe double-check on your end if there is something else needed to be able to access the organization. :slightly_smiling_face:

[danielquinn]

Thanks for noticing @pitkley. I've just now approved the Travis CI integration. but it doesn't appear to be working. Details are in #512.

[ovv]

@danielquinn I was looking into moving ovv/ansible-role-paperless to the organization but I don't have the right to create new repository (required to do a transfer). Would you be ok to grant me that permission ?

[danielquinn]

Yes I would be happy to grant you that permission, however I'm not 100% on how GitHub does this stuff. I've now created a team called Reviewers and you're all on it. You all have Manager privileges, which I think gives you the right to create new repos in the organisation. Have a look and see if you can create a new repos, and if not, I'm going to need some advice on this.

[ovv]

Doesn't look like I can. I went looking in another organization and I believe you need to tick: settings > Members privileges > Repository creation > Public and private repositories

[danielquinn]

Ok! Give that a shot now.

[ovv]

Looks good. thanks :+1:

[danielquinn]

Hi guys, just a heads up about the Docker stuff. Currently, the "official" Docker image is at danielquinn/paperless but I think it's probably time to change that to be thepaperlessproject/paperless. I'd wire everything up myself, but they don't appear to allow for automatic (or even manual) builds from Github without giving them write access to all repos. I'm not going to do that, but if any of you are cool with it, I can make you a manager on the Docker hub group (I already promoted @pitkley and @jat255 as they were the only reviewers I could find by username so far). If one of you can wire up this repo to that project, I'll kill the old one (there doesn't appear to be a way to transfer repository ownership on Docker Hub).

[pitkley]

@danielquinn unfortunately it doesn't look like I am able to link the paperless-org or the paperless Docker repository to GitHub, I do not see the "Settings" sub-menu where it would usually be:

[LorenzBischof]

@danielquinn I recently setup drone.io to build me an armhf Docker image. If you don't want to give Dockerhub full access, then this could be a solution? https://github.com/lbischof/paperless/blob/armhf/.drone.yml Using something like this: https://docs.drone.io/user-guide/pipeline/multi-platform/ it would be possible to have Images for every platform.

edit: I tried to find out what permissions drone requests. Instead of: "Full control of private repositories" (Dockerhub) it only needs: "Access public repositories". The wording makes me assume this is read only access.

[danielquinn]

@pitkley I poked around Docker Hub some more and found the Permissions tab, where I assigned read/write permission to the managers team:

Maybe this does it?

@lbischof drone.io sounds nifty, but I think if I wanted that degree of control from the repo-end (as opposed to Docker Hub's pulling of the code and the resulting "automated build" status), then I'd probably just use GitLab to do the work. Here's the output of another project I'm working on over there -- scheduled automated builds in multiple environments etc. (.gitlab.yml). I'm not sure it'll do Arm, Mac, or Windows though. If Drone can do that, it might be handy to keep in mind.

[sbrunner]

What the status of the Docker image? Is some help needed?

[ddddavidmartin]

What the status of the Docker image? Is some help needed?

I believe the automated build is not working and it is somewhat out of date. I don't quite remember who was last looking into it though.

[danielquinn]

Until we can figure out how to let Docker do an automated build, I'm afraid the only option is to do the build yourself with docker build . -t paperless. We could automate the push to Docker via Travis, but that wouldn't be an "automated build" according do Docker, which is less desirable.

[ahyear]

is it the same than docker-compose build ?

[danielquinn]

Sorry @ahyear, yes it's the same thing. docker-compose is just a smart wrapper around docker that automates the process of standing up an image into container along with all of the other configurations we need to make use of that container.

[sbrunner]

I just open a Pullrequest for a proposal to deploy the Docker image ... https://github.com/the-paperless-project/paperless/pull/532

[pitkley]

@danielquinn unfortunately the change to the permissions didn't seem to have any effect, I am still unable to configure the repository. :slightly_frowning_face: What I did though is try to request access for the "Docker Hub Builder" app to the-paperless-project organization. If you can approve that (you should have received an e-mail), maybe that helps in solving the issue.

[danielquinn]

I've just approved "Docker Hub Builder" so hopefully that'll get things rolling. If there's anything else, just ask.

[ddddavidmartin]

Hi all,

https://github.com/the-paperless-project/paperless/pull/532 has been merged in but I don't think any builds have made it to Dockerhub yet. Is this something for @danielquinn to arrange?

It would be great to get this going as the current official build still has the convert error that stops documents from being parsed.

Thanks everyone for looking into this!

[danielquinn]

From the looks of #532, my understanding is that we now have a way of making Travis push updates to Docker Hub, and all that's required is that someone set DOCKER_USERNAME and DOCKER_PASSWORD into Travis so that those values are available to the deploy script.

So as a test, I went ahead and added these two values to the Travis secrets and re-ran the most recent build. The build script exited with a 0 status, but nothing appears to have been run :-(

Additionally, this isn't a good permanent solution, as anyone can get my docker credentials now simply by pushing a commit that says something like:

echo ${DOCKER_USERNAME}, ${DOCKER_PASSWORD}

...so I'm going to delete these secrets from Travis before something like that happens ;-)

We could have a role account -- one whose only powers include writing to the Paperless organisation, or someone with admin rights to this repo on Docker Hub could add Github integration, though there are some scary security concerns with that option (which is why I haven't done this myself).

I'm open to suggestions here.

[sbrunner]

... anyone can get my docker credentials now simply by pushing a commit that says something like ...

This is wrong, on a fork the secret will not be set, then this is true only for committers.

Event if I think that's good to have a technical account on Docker hub only for that :-)

[danielquinn]

I'm not sure I understand. Surely, if we create a role account and set the user/pass into the Travis secrets, those secrets could be exposed if one were to echo them in the ci/deploy-docker script. Otherwise, they couldn't be used in said script.

[sbrunner]

If it's pushed on a fork it can't be echo on the and the ci/deploy-docker can't works because the variable are not interpreted.

If it's pushed on the the-paperless-project we can do an echo and the ci/deploy-docker will work but only committers can do that (not anyone), We can say that's not completely safe but it's not so simple to get the secrets :-)

[danielquinn]

How 'bout this:

1. I create a dummy role account on github, something like paperless-dummy
2. I then add said account to Docker Hub as one of the contributors and use it to automate the build process.