Discuss future plans for Trovebox

[jmathai]

Things have been quiet around here for a while. It's due to multiple reasons including the core team focusing on the business side of things and then ultimately the shutdown of Trovebox.com.

Quite a few folks have asked what will come of the Trovebox community and open source software. This is where we have that conversation. What you see here is the result of several years of full time commitment by a small team of folks plus contributions from the community.

Trovebox exists as an organization on Github and we have 25 members with various levels of access to the repositories. We should revisit the different teams and see who wants to remain an active participant and add any who are not yet involved.

The plan for features and direction was in the past directed (almost entirely) by the core team. I'd like to keep the focus narrow and spend time on polishing current and new features instead of allowing lots of sub par features to creep in. That's been one of the successes of Trovebox thus far.

That being said, let's discuss.

[rufuspollock]

Hi, I'm Rufus Pollock from Open Knowledge.

I'm delighted to see this issue as I was wondering what are your plans for stewarding the code and (as a distinct thing) the service going forward? (By service I mean a (central) hosted instance of the code that is usable - i.e. a continuation of https://trovebox.com and its relevant APIs).

I ask this both as an avid user (I am uploading photos from my phone yesterday) and as President of Open Knowledge where one of the major things we think we could (and should) be doing is stewarding important open data / open content / open service projects (in a way similar but different to the way Apache acts for certain open-source software projects).

Trovebox, whether in terms of code, or even in terms of service, seems to be something that is eminently worthy of this stewardship. You may already have plans here but if you did not I wonder if it would be worth a quick chat to see if there were any possibilities where Open Knowledge (and more importantly its community) may be able to contribute in some way.

---

A end of service question: whilst raw photos have already been archived to our chosen storage what about the metadata (tags, creation dat etc). I believe that #1322 never got implemented (#1322 = Store metadata with photo in backend storage (as well as in DB))

[jmathai]

any possibilities where Open Knowledge (and more importantly its community) may be able to contribute in some way.

@rgrp can you expand on this? Has OKFN worked with other communities or projects to help in a similar transition?

You're right about #1322 never being implemented. This is something we've had internal discussions about for a long time. I would like to incorporate this into the archiving but it seems to make more sense to be added to the core code (and by extension, the archiving). There was never a pull request for it or any code that does this work so I'm open to ideas.

[michielbdejong]

About the service part going forward, I'm offering continued Trovebox hosting for 8 euros a month. So far, two people have signed up (although I have yet to import their data, hopefully that will succeed in the four weeks left before the 15 March shutdown).

I would also be interested in continuing hosting for all customers (how many are there, actually?), simply to stop this content from disappearing from the web. Maybe OKFN (and/or the Internet Archive) can help with finding the necessary money for that (if I offer the sysadmin work as a volunteer, then what would ballpark monthly hosting costs be?).

It would be great to send out an announcement one day before shutdown-day that trovebox.com URLs will stay up, thanks to a coordinated effort of the community. In particular, it would be a token victory for the open web, and could be repeated at the next site death, provided the code is either already open source, or open sourced at shutdown.

[michielbdejong]

@jmathai this is starting to get very urgent!

On https://trovebox.com/ it (still) says you’ll be shutting down the hosted Trovebox service on March 15, meaning we only have little more 3 weeks left to organize all of this, right?

[michielbdejong]

Got a reaction from @jmathai via Twitter two weeks ago, but no movement on any of this since then.

Let's hope they don't shut down any of the accounts without resolving this. For now, it looks like I'm going to have to issue refunds to the people I'm trying to help to move off trovebox.com. :(

[jmathai]

There hasn't been much feedback in terms of ideas of volunteer efforts to continue Trovebox. After much thought and discussion I believe it's best that I'm not personally involved after any possible transition. It seems that will help Trovebox evolve on its own; and that makes sense to me.

@rgrp Could you elaborate on what OKFN could do to help?

@rgrp: what about the metadata (tags, creation dat etc). I believe that #1322 never got implemented (#1322 = Store metadata with photo in backend storage (as well as in DB))

In lieu of implementing #1322 we did make sure that the exports store tags, titles, descriptions into EXIF/IPTC. This was confirmed in the last email we sent out.

@michielbdejong: no movement on any of this since then

@michielbdejong We've sent out an email last week. Admittedly, the progress has been slower but our goal is to make sure users get their data out. I'd prefer to keep this thread focused on figuring out what happens with the Trovebox source code. I did updated website though.

Trovebox.com site will be shutting down March 31, 2015. We may extend this deadline to help accomodate customers to obtain archives of their photos.

[russmac]

Hey, You may not have noticed but your project appears to be non functional You have several tickets here indicating that regardless of configuration your setup does not work.

Could you update your readme.md to indicate its broken or provide a base sql for people to set things up themselves. Otherwise many more people may sink time into this without knowing its broken.

[russmac]

See #1572 I have engineered a work around for you.

[hfiguiere]

I opened #1584 as a way to initiate a "new governance" for the project. By governance it is about put the open source project back on its feet so that it can: 1. deliver software 2. accept contributions.

For 2, we need a clear contributor policy, including, I highly recommend, a process of signing off on pull request (reviews)

[jmathai]

Agree with @hfiguiere. After discussing transitioning Trovebox with others I believe it's better if I'm not actively involved once the "new governance" is set up. However I'm happy to participate in that transition.

We need folks to voluntarily step up to take the reigns though :).

[rufuspollock]

@jmathai couple of thoughts:

• Could you do a call out for maintainers through existing twitter etc pointing to e.g. this ticket
• Would you also be transferring things like twitter and any other relevant assets to group of maintainers

For my part, I note that you may want some institutional legal home - if only to do things like be clear on who can act as legal custodian for assets including accounts and code etc. As I've mentioned above Open Knowledge have been involved in this way and as general contributors in various projects such as http://ckan.org/.

[jmathai]

@rgrp Done and yes. Happy to transfer all accounts and assets. Would prefer an institutional home as you suggested and would love for it to be OKFN :).

https://twitter.com/trovebox/status/621724451453841408

[emanchado]

@jmathai will you guys be able to answer questions about the code? I think that could be extremely helpful. Maybe create some kind of mailing list or forum?

I certainly won't have time to be a maintainer, and I'm not even sure I have enough interest, but I might be able to help with a ticket or two, esp. if I feel I don't have to figure everything out myself :-)

[jmathai]

@emanchado There's a mailing list but most discussions have happened here on Github or Twitter. The mailing list is there though and people are subscribed. Unsure how effective it is.

However questions need to be answered we can make sure questions are answered and documented.

[httpdispatch]

@jmathai please accept new google play developer policy so the trovebox android app may return to the google play store. Also i would like to update android app with basic offline support.

[Yetangitu]

I'm getting close to a working PostgreSQL port for trovebox/openphoto (perhaps it makes sense to go back to the latter name...?). Once this runs as intended I intend to integrate it into Owncloud in some way, either by implementing Owncloud as a backend storage or by creating an Owncloud app based on trovebox.

While creating this port I noticed - by virtue of PG being more picky about such things - quite a few what I'd describe as 'mysql-tainted' anomalies, ranging from violated constraints (eg. elementtag.actor being NULL even though the schema says NOT NULL, etc) via odd character-case related problems due to camelCased columnNames bleeding through into array keys to 'abstractions' which end up codifying mysql-specific traits (eg. EpiDatabase relying on driver-specific PDO::lastInsertId() behaviour). Fixing all this will touch a rather large part of the code base, not limited to the database driver. It would make the thing a bit more robust, even for those who don't care about PostgresSQL or other databases.

Quite a bit of the code seems to be related to features which are not available in the free version. Is there any chance of that code ever being made available? If not it probably makes sense to cull the remainder as it won't be of any use.

[rufuspollock]

@Yetangitu this is great to hear. I'm really keen to see this back up and running.

Aside: I still think we probably want some entity or group that act as custodians for the project and codebases etc. I reiterate that Open Knowledge, via its Labs would be happy to be a legal, institutional home with a group of specific people acting as the actual project maintainers.

[Yetangitu]

Well, I've had it running for many years now (using mysql/mariadb and lighttpd) so in that respect my work does not add that much. It is still possible to take the code from here and get it working, with a few adjustments here and there. I also use the Android app to feed the service, this mostly works as intended although the app could do with some TLC when it comes to selecting which images to sync - wading through that list of ALL images found on the device (not just photos, but anything resembling an image) is tedious at best, impossible in practice.

It would be beneficial to have the code live in a less moribund backstreet of the 'net, yes. I'd rather discuss changes in a lively atmosphere instead of in a morgue where I have to suppress the urge to whisper...

[jmathai]

@rgrp what's needed in your mind to transfer the legal and institutional home over to OKFN?

Anyone on this thread interested in being a project maintainer?

[Yetangitu]

@rgrp is there anyone in Open Knowledge who feels the urge to answer @jmathai 's call for a project maintainer? Did you have any specific people in mind when you mentioned that 'group of specific people acting as the actual project maintainers'? I'm asking because I'd like to see this project land somewhere safe before all those who show any interest have left for greener pasture. If you can not find anyone I can take up this task for the time being, but not being part of OK and already active in several other projects I don't want to promise more than I can deliver.

[rufuspollock]

@jmathai @Yetangitu thanks for pinging - i have been in action at this end. To be clear I don't think anyone at Open Knowledge would be project maintainer - rather one would want to establish a group of individuals willing to commit to do that. This would be the place for people to volunteer - I also emphasize that one could also do a proper call-out on this once one had one person willing to act as initial coordinator (I'm not sure that many people read this thread).

What Open Knowledge can do via e.g. Open Knowledge Labs http://okfnlabs.org/ is provide a secure long-term legal, social and technical home. There is also a part-time labs coordinator who can provide some assistance e.g. in recruiting initial maintainers.

[youphyun]

It is silent here in this threat for 2 Months already. Isn't is better to try to join the forces together with the ownCloud Gallery+ app? I can even see a nice trovebox screenshot there: https://github.com/owncloud/gallery/issues/255

Both ownCloud and the Gallery+ app are actively developed and it is also possible to support new features using Bountysource.

[jmathai]

Being the pragmatist that I am I don't know if there's enough interest/momentum to really keep this software moving forward. ownCloud is a reasonable alternative so maybe we change the discussion to seeing if there's anything from Trovebox that could benefit ownCloud or another alternative.

I don't want to keep a project alive just for the sake of it. I think there's a lot of important ideas from Trovebox that are unique to this project and maybe it's a better idea to translate that to another project with more momentum.

Any convincing arguments otherwise?

[Yetangitu]

Just brainstorming here so don't trip over missing bits...

Moving the salient bits from Trovebox into an Owncloud app might be a good solution to fit the needs of those looking for a self-hosted photo sharing site. The current Owncloud apps which partly cover this area (Gallery and Gallery+) are not up to this task due to a lack of performance and features, especially when presented with a 'large' image collection.

I currently use Trovebox (both the original as well as the PostgreSQL port) for this purpose, keeping the entire photo work-flow outside of Owncloud due to the mentioned performance problems. Part of those are caused by my use of older hardware (Intel SS4200 servers with Pentium E2220 cpus) but this is a conscious decision - software like this should be usable on lower-spec hardware, people will be running this stuff on Raspberry Pi boards and equivalents. Trovebox works in this situation, Owncloud itself also works (sorta) but the current Owncloud gallery apps don't cut it.

The ideal solution would be to have the-system-formerly-known-as-trovebox as a frontend, with the content managed through an Owncloud app using existing and to-be-developed Owncloud functionality to feed images (and potentially other media like video, audio and documents). The actual presentation frontend - which faces the outside world - would preferably be only loosely connected to Owncloud for performance as well as integrity/security reasons. Owncloud is big and growing bigger by the day. Given the way it is used, it has access to privacy-sensitive data. It already offers a file sharing mechanism which could be used as a backend feed for Trovebox, similar to the way Trovebox can use Dropbox et al. This way the attack surface for Owncloud would not be increased and it would be easy to run the frontend on another machine like that Raspberry Pi I mentioned.

Implementing the first stage of this merger would entail writing an Owncloud storage backend for Trovebox. This could be followed by an Owncloud app to manage one or more Trovebox instances, feeding data to those through the Trovebox API. This would allow the use of existing Owncloud syndication features like file sharing, replacing the (rudimentary) multi-user functionality in the publicly available version of Trovebox. Once Owncloud gets a working and usable tagging system this could be used to create tags (obviously) and albums in Trovebox. Later on more functionality could be moved from Trovebox to Owncloud, eventually leaving Trovebox as a presentation frontend for Owncloud-hosted content. This assumes that the Owncloud sharing subsystem is able to deliver those previews and images at a reasonable rate, something it currently has problems doing - at least on the mentioned lower-spec hardware. For now I'd keep the gallery preview images under Trovebox' custody, feeding only full-size material through Owncloud.

While all this at least initially leads to some duplication of data and functionality I'm willing to live with this compromise in exchange for better performance and a lower attack surface.

Now all that remains is to implement it :-)

[oparoz]

The current Owncloud apps which partly cover this area (Gallery and Gallery+) are not up to this task due to a lack of performance

That's not entirely the app's fault, but partially a problem with ownCloud's core, so you're going to experience the same problems with an external app. There is a huge difference between Trovebox' file management features and the ones of ownCloud and ownCloud takes some tuning to get it right.

Regarding the large collection argument. If your collection constantly changes, then yes, generating thumbnails takes a lot of time on less powerful hardware, but otherwise, you can simply generate your thumbnails once and the Gallery app will fly. The demo proves that. In any cases, there are designs available to fix all sorts of issues with thumbnails (standard sizes, pre-generation at upload time, meta-data storing). They're not huge tasks and they're just waiting for coders to implement them.

By using Trovebox as the front-end and thus as a new caching layer, you may improve performance locally, but mobile apps connected to ownCloud, and later, other users of the system (shared thumbnails cache) won't be able to use that cache. It's just one of the reasons I think it's better to fix ownCloud unless changes are rejected in core, but both Gallery and Trovebox can evolve in parallel, stealing ideas from each other and hopefully fixing problems in core.

[Yetangitu]

You're thinking in Owncloud terms - mobile apps connected to Owncloud, other users on the system, etc. I'm thinking in web terms - anonymous users on the web, browsing images. These are two different use cases with different demands.

To put this difference in perspective consider that a static site generator can be used to serve the latter purpose - and indeed is often used for that purpose. As far as I'm concerned the only reason not to use a static site is the large and increasing update cost for larger sites, this being the reason I went from using static sites to Openphoto -> Trovebox.

In my perspective the front end experience would not change radically, other than fixing some nagging bugs. The back end, the feeding of the thing, would be managed through Owncloud. For that purpose the somewhat lackluster performance offered by Owncloud does not matter that much.

Given time it is certainly possible for Gallery(+) to close the performance-gap but that still leaves the question whether it is desirable to open up an Owncloud installation to the whole world like you'd do with Trovebox. I'd rather keep these systems separated, at least for now.

[oparoz]

I'm thinking in web terms - anonymous users on the web, browsing images.

That's only one use-case of Trovebox. Editing the title or tags and uploading pictures are not or rarely are public tasks.

Besides, ownCloud's Gallery does have an anonymous side, available via shared links, so I'm not sure what the big difference is here (except the one where you expose your ownCloud installation to the world). Once thumbnails are generated, it takes about 1s to get the list of images contained in a folder. I don't think we can consider this to be poor performance.

it is certainly possible for Gallery(+) to close the performance-gap

Now that I think I understand what you're trying to build, this is a side note. You haven't described what the gap is, where it comes from, etc.. It may be that there is no gap for most people willing to perform a few extra steps or a huge one which has not been documented yet.

whether it is desirable to open up an Owncloud installation to the whole world like you'd do with Trovebox. I'd rather keep these systems separated, at least for now.

OK, so what you want to do is duplicate your media collection. One system will manage the files, the other will be used to publish them on the web.

You won't be able to use the ownCloud cache as it may be incomplete, so you'll need to copy each full image and will only be able to see pictures, not media files (without writing your own converters)

[oparoz]

The ownCloud app pushing changes to Trovebox could simply be an occ command. You would generate thumbnails and push IDs to Trovebox. That way you could share the cache instead of having to duplicate the media collection.

[Yetangitu]

Storage is cheap, duplicating part of the cache - and it will only be that part which is meant for public consumption - does not bother me. The public-facing Trovebox instance would only see those images which are meant to be published, in my case less than 5% of what is available.

The bit about 'editing the title or tags and uploading pictures' are just those tasks which would eventually be moved to Owncloud in my 'brainstorm' system, leaving Trovebox as a presentation frontend.

[ this section should be discussed in the Owncloud thread, but since you mentioned it...]

The performance problems I've encountered with Owncloud gallery applications generally center around image discovery. It can take a long, long time for anything to happen once the app has been started. In the background the app is going over each and every potential image in the OC user dir, grinding away until it eventually produces the desired overview. Once it has gathered all images performance is on par with Trovebox but I usually do not have the patience to wait for it to get that far. If this long time-out for image discovery was a one-time thing it would be bearable but it is not. This delay occurs each and every time Gallery(+) is started. This despite judicious use of .nomedia to limit Gallery(+)'s inquisitiveness to a subset of the available (large, ca. 3 million files, 3.5 TB) data set.

Gallery(+) also lacks some features wrt Trovebox but those could be implemented. My main concerns with using Owncloud for the tasks currently covered by Trovebox are the already mentioned performance issues and my hesitance to expose the big (and growing) elaborate mass of code which is Owncloud to the whole.wide.world. I prefer to use tailor-made tools for specific tasks over 'systems' which provide all functionality under one umbrella (and yes, I generally prefer vi over emacs, even though I use the latter for some tasks...).

Now that I think about it, maybe I should just develop an Owncloud-managed static site generator based on one of the many specimen of such out there...

[oparoz]

Sorry for the long delay in replying.

leaving Trovebox as a presentation frontend

OK. This would kill Trovebox as a photo management system and turn it into a static generator for ownCloud Gallery via its own ownCloud connector. I don't think this would necessarily be a bad thing as this would make Gallery better, adding all the existing features of Trovebox and would provide speedy browsing for people who have rather static image collections.

The performance problems I've encountered with Owncloud gallery applications generally center around image discovery. It can take a long, long time for anything to happen once the app has been started.

This happens if you don't use Redis as the caching mechanism on 8.1+, as explained previously. It takes 35s to discover 27 images if you don't enable it. With Redis, it takes 2s, making it usable in most situations.

[ghost]

L

[hfiguiere]

Here is the thing. I'll have some free time in the following weeks and here is what I sort of intend to do. If you agree, it will happen here, if not, it will be my personal fork.

1. create a new master branch for the community version. If master is suitable, then master it will stay. The idea is to have something that we can work off.
2. start prioritise issues for a the first "community" release. with what needs to be done.

[hfiguiere]

1. accept PRs :-)

[hfiguiere]

Also I have concerns about the name Trovebox, its domain name, etc. The transition to a community maintained project require stewartship of the trademark (if any) and domain names. I know the domain trovebox.com is still registered to @patricksan but there is nothing at the end.

[oparoz]

@hfiguiere - Domain name, code, mobile apps and community are all dead, I think it would be easier for you to just fork it, clean it up and to try to build a new community by pinging people when you're ready, but to be honest, unless you manage to find 2-3 people to work with you and some backing by an organization, your fork will probably suffer the same fate as Trovebox, unfortunately. Maybe a new crowd-funding campaign could help bring the app back to a good state.

[jjdelc]

After the shutdown I initially decided to just run my own copy and carry on developing on a personal for, but had trouble even starting the installation and poor docs.

The killer feature for me was the phone app.

Sadly, after all the trouble I just decided to code my own solution https://github.com/jjdelc/Photolog

[jmathai]

@hfiguiere thanks for chiming in. I would be happy to assist you in fixing any bugs and starting towards a new version. I'll ping @patricksan about the trovebox.com domain which we've keep registered because we have email addresses there which now forward to personal accounts.

I'm also happy to help prioritize the backlog of issues.

Lastly, I would want to hand over the keys.

@oparoz I agree that the community is dead but the domain name, code and mobile apps still have value. I imagine that @hfiguiere would have a more difficult time building a community around a personal fork than through this organization. Building an open source community isn't easy; no matter how you slice it.

@jjdelc the mobile apps should all work with any API compliant service. The API docs should be fairly thorough as others have built their own apps off them.

[hfiguiere]

@jjdelc The code for the Android app is there: https://github.com/photo/mobile-android - no sure how it matches with what's published in the Play store. The iOS one is https://github.com/photo/mobile-ios So they can be maintained.

@jmathai I'd love to get the keys.

So I should start scrubbing trovebox.com from the code then (at least as links).

[oparoz]

Thanks @jmathai. Good news that you can hand over the keys.

• The code is of good quality, but quite large, designed for PHP 5.3 and all custom made (no composer), so it will be a lot of work to modernise it and won't take just a few weeks
• Mobile apps quickly look out of date and stop working (or become insecure) when not updated for 2 years, so lots can still be saved, but developers still have to be found, unless 3rd party mobile apps have taken over
• Building a community is hard, but trying to interact with one built by someone else who isn't present any more is harder, unless you become the new maintainer. Good thing you gave a sign of life :)

[jmathai]

@rgrp Would OKFN still be interested in or a good home as a foundation for Trovebox assets? If so please point someone from there to this thread (since you've stepped down).

@hfiguiere On the mobile app side...the apps that are in the app stores include a screen to log into your trovebox.com account. The ones on github are otherwise identical and require you to follow the standard oauth flow.

On the iOS side there's also the Delightful app which @nicnocquee agreed to open source. It's pretty fantastic.

You're an admin of the organization. Do you need anything else besides things like domain (which I need to talk to @patricksan about)?

@oparoz I agree WRT the code. It's a bit old and crufty but it works and the bugs are known. I won't speak to it but Joel Spolsky articulates the loss of value when throwing away code that works.

There are no devs in line to work on the mobile apps. The Android app was built mainly by a contractor and the ones working on the ios app have moved on. Perhaps they're interested in reviving the app but I wouldn't know.

I originally wanted to be involved in the long term plans for Trovebox but after speaking with several folks who were the main leader for open source projects they all suggested that I should step aside. I think @hfiguiere is a good person to take over the reins and his ideas probably differ from mine. My presence would make the roadmap murky and confusing. Plus he's been doing open source for a long time and I trust his ability and commitment.

It's sad to not be actively involved (see my first comment where I explicitly say I prefer to be involved). But I think the folks I talked to are right about what's better for the project.

[patricksan]

@jmathai @hfiguiere no Problem at all for me. Just ask what you need and I will do.

About the iOS code if needed I can help again. It doesn't take that much time. As @jmathai said there is also another app - https://github.com/delightfulapp/delightful - it is a more modern approach but the one I did is more complete. I can contribute back to the iOS if the community is back.

[patricksan]

@jmathai @hfiguiere for Android the best is create another account on Google and deploy again.

Both apps need some refresh.

[rufuspollock]

@jmathai OK Labs I think would be a good home (with OKI as legal entity). I am still involved as a community member with Labs and have raised and issue here:

https://github.com/okfn/okfn.github.com/issues/426

Comments and thought here or in that issue are welcome.

Great to see movement here :smile: