Future of the project / collaboration

[adam-stanek]

Hey @dskaggs, hope you had a nice Easter break.

Some time ago I was thinking about becoming contributor to the Lodestone project. I have written pretty much similar application few years back. It was used for photo organization, but died in the drawer of unfinished projects. Later I dived into the world of document management and I realized it is the same thing over again. I was thinking about how to not reinvent the wheel and maybe create application which would be suitable for both. It is all about file indexing after all, the only difference is what kind of metadata we index.

I was trying out Lodestone and thinking about possibility of extending it for additional use-cases. I really liked the idea behind it and even though it lacks some features that other solutions have, it was a clear choice for me. But as I dug deeper I discovered that a lot of the components are more in the state of proof of concept and some of the design would have to be changed in order make it work.

I have started learning Go few months back and took the opportunity for a new pet project. I have rewritten the whole thing from the ground up. I still followed a lot of the same principles as Lodestone but went further into making it extendable and configurable for different cases.

I would like to see it as a platform which gets DMS functionality out of the box but also offers way how to easily plug in custom processing solutions. There is a lot of guys out there experimenting with machine learning stuff. I wanted to create a platform which would help them focus just on the cleverness of their solution instead of file indexing / processing.

Right now I am at the state where I have most of the backend and processing stuff in the same / better scope of functionality as Lodestone but I have not started on the UI / Frontend part yet.

I remembered that you mentioned that you are a primarily frontend developer. Since both projects overlap each other would you be interested in joining the effort? I would gladly cover the BE part to push the project to the production ready state.

Have a look: https://gomimir.gitlab.io/

Cheers,

Adam

[dskaggs]

Wow! That looks incredible. The tagging rules engine is specifically something that I had hoped to add to Lodestone at some point in the future. I also like the ML possibilities as well as I think that would be an amazing addition if we can make the administration of it easy for end users.

Were you thinking of continuing with Mimir as a separate, standalone project complete with UI or combining the 2 projects into one? From what you've said and after looking over your docs, so far Mimir is a reorganized and upgraded version of Lodestone's backend components. Does it make sense to start a completely new project, or join the two together and take advantage of the (admittedly small so far) community that Lodestone already has?

[adam-stanek]

Thanks, I am glad you like it. UI for the rules will surely be a challenge :)

Honestly, I haven't got that far to think about the project organization just yet. I just wanted to reach out and put it out there for discussion :) I think I am ok with both variants. For me the most important thing is for it to start getting some traction. It is big project and we will need to build active community if we want it ever to reach its potential.

I don't really mind if it is going to be called Lodestone 2.0 or Mimir or whatever. But I put decent effort into setting up the project on GitLab with all the CI / integrated package repositories goodness coming with it. So I would rather if it stayed there as I don't have that much experience with GitHub actions or willingness to migrate it just for the sake of it.

[dskaggs]

I can take care of the automation pieces. That's one of the areas that really interest me (and honestly I'm better at that stuff these days than I am at writing Angular code). Let's definitely keep talking and see where things go.

[adam-stanek]

Ok, fair enough. But for now, migration itself won't really help us move forward.

For me the biggest question to solve is how to bootstrap the FE development. Without UI it has no way of reaching wider audience. We are pretty much looking at a new codebase giving the fact how much the API changed with Mimir. I can surely start but I was hoping you would drive it. There is still plenty BE stuff for me to cover.

Now that you mention Angular, it is probably the right time to ask - I am actually a TS/React developer as my main profession. I initially thought I would go the React way ™️. Would make sense to me personally, given my background, and the fact that I think we could reach maximum amount of possible contributors that way. I am not set against Angular, but it would mean you would have to be the one to drive that forward. I am sure I would be able to contribute in some way, but I would definitely not be proficient enough to set the project course with the best practices in mind.

[dskaggs]

Ah, the age-old debate (at least in front end development) :) I actually know next-to-nothing about actual React development. For our needs at work, the Angular ecosystem was the better approach to support the dozen plus teams that need to work in our monorepo. Obviously any technology can be good or bad depending on the engineers writing the code and no one tech is the right answer for everything. I do have a fundamental dislike of the idea of JSX though :)

[adam-stanek]

Uff, that kind of puts us in the tough spot 😆 You know what, it would not hurt me to learn a bit of Angular. I have nothing against it personally. It's just as you mentioned, React was the way I have chosen for projects at my work :) But one day you will have to tell me, why from all the things you don't like about React is JSX the one which hurts you the most ;)

[dskaggs]

Hahaha...fair enough! DM me on Twitter (same username) or via the Gitter link in the Readme and I'll tell you (so I don't get hate mail here).

[James-Firth]

Hi @dskaggs and @adam-stanek I was just looking at getting a nice home server set up recently and stumbled across Lodestone

Has most focus moved on to Mimir at this point on Gitlab or was there still thoughts of merging, etc? I'm eyeing the repos and seeing a bit more activity there but want to know where I should be looking :)

[dskaggs]

@James-Firth Honestly, I think when I agreed to take over as maintainer on this project, I had a "my eyes are bigger than my stomach" moment. I've been remiss about moving it forward. @adam-stanek helped out a bunch on the Go services some time ago but I just haven't had the time to put into Lodestone. I've actually been experimenting with PaperlessNG in my home lab and am probably going to go that route for my uses.

[James-Firth]

@dskaggs Totally fair, I've jumped on my fair share of projects with good intent but not enough time! (I'm a team lead and sometimes even get to write code as a fullstack dev sometimes!)

I know the original author mentioned PaperlessNG messes with the files, any chance you could elaborate on what it does? I initially saw PaperlessNG and was thinking of it for my homelab, but liked the UI of Lodestone more (despite seeing the deprecation etc)

I'll have to either check out @adam-stanek's Mimir when it's ready for use or check out PaperlessNG myself :)

[adam-stanek]

Hi @James-Firth, I did decent amount of work on Mimir, but I think it is still a few months away from prime time. It is completely new platform, not a Lodestone evolution. It builds upon similiar principles - Elasticsearch and S3 backend on microservice architecture, imagick + tika for processing under the hood. But my ambition was much bigger then DMS. I am building general file indexing platform. It does not matter if you are indexing documents, your photos, videos, or whatever files you have. From each you can extract the metadata and search through them. I was aiming for making the whole thing extendable so that it is very easy to write your own processor for specific files you might have (there is an exposed API + library for that). It is built around of set of rules of what the system should do with individual files.

From the end-user features:

• Multi-level tagging (you can have different types of tags)
• Tagging and processing based on file path - you can leverage your existing directory structure and tag the files by it or process them differently based on their file location
• User defined properties - you can define whatever you want for your files - ie. due dates for invoices
• Faceted search - you can narrow down your full-text searches by tags and properties
• Downloading results of search as ZIP archive
• Sharing links to download individual files
• Native upload directly in the app
• Authentication - integration with proxies and OIDC
• Index level permissions, document / tag level probably later this year

Majority of that stuff is fully implemented on backend, but I am only slowly getting there on the frontend part. The thing which is holding it back the most is any frontend for the configuration, which makes it quite hard to start with in the current state. I would suggest to check back in few months unless you want to make your hands dirty :)

[James-Firth]

@adam-stanek (whoops this sate in drafts all day) Thanks for the info! Yeah I can see this expanding to more than DMS. It seems modular though so if someone was focused on DMS they could "just" use those features right? Love the extensible idea.

For the UI are you thinking of also going modular with a "documents tab" and a "photos tab" etc? I could see photo management being a very different UX for instance.

I can't promise any time but are you looking for help via issues or PRs or anything? I'm intrigued by the backend languages but I do React work when I'm wearing my developer hat at work.

[adam-stanek]

@James-Firth Yes, it shall be like that. It is up to the user to choose what functionalities fit his needs.

Regarding the UI, this is very good point. I was thinking about it somewhere along the lines how Google Drive / Docs does it. You have one lets say more generic approach (Google Drive), showing you all the possible files with timeline, full text search, etc. However you would also have specialized views (Google Docs, Google Sheets, ...) showing you a tailored perspective - subset of the files. For example for photos I imagine this would be primarily structured along albums and identified people, objects, places, ... Under the hood, the BE part, is pretty much agnostic to it. It is all build around tags and properties. But on FE I believe we really need to tailor it for decent UX. Mimir has it in form of configurable indices. I am expecting that you can just choose what "view" fits your case for each index.

I welcome any contributors, both BE, FE or even people interested developing individual processors :) The FE is Typescript + React + Chakra UI if you are interested.

[James-Firth]

Thanks for the discussion you two! I'll keep Mimir discussion on the Mimir repo and Lodestone info here if I do any more poking around :)

[fivestones]

Did anything ever come of this? It seems like maybe Loadstone is dead at this point, since @dskaggs has moved to paperless-NG?

One thing (among others) that Loadstone offers which isn't available in Paperless-NGX is that it leaves files alone on the disk instead of reorganizing/renaming them like Paperless-NGX does.