hledger-web on sandstorm is not satisfying [$100]

[simonmichael]

Negative feedback is accumulating at https://apps.sandstorm.io/app/8x12h6p0x0nrzk73hfq6zh2jxtgyzzcty7qsatkg7jfg2mzw5n90 , need better Sandstorm integration to fix this.

[ocdtrekkie]

Another note (for myself) is that next time I push an update here we should include screenshots in the Sandstorm metadata so that people can see what to expect when they install the app. Also, the screenshots will provide another way to give an example of what the app's data entry should look like.

[simonmichael]

+1. Great work, thank you! And adding screenshots, or a screenshot-based mini-tutorial, is a great idea.

[ocdtrekkie]

So my big question is: What is the expected workflow of an average hledger-web user? Since it's currently view/add-only, do we assume people will maybe run it on a PC they have terminal access to, and do more intensive ledger operations when they return home/work?

I am wondering how much effort would be entailed in adding a basic "Import Ledger" and "Export Ledger" button set that overwrote the ledger file from an upload, or offered up the ledger file for download. Then users could, say, export to whatever PC they're working on, carry out their ledger operations locally with a CLI, and then re-import it to store on their server.

(I know you can use grain backup and restore to hackily do this on Sandstorm, but there's a lot of problems with that, since restoring creates a new grain, rather than updating an existing one. If you share it, you'd have to share it anew with everyone each time you did this.)

[AaronM04]

Hi @ocdtrekkie , speaking as a hledger-web-on-Sandstorm user, I can say that "Import Ledger" would be very useful. "Export Ledger" is not that big a deal for me, because I can click the backup grain button in Sandstorm and then run a script I wrote to unzip and copy the ledger file over.

An import button would save a lot of work, since there's currently no way to do this other than restoring a grain, as you described.

[ocdtrekkie]

@AaronM04 That's good feedback to hear. Note that export is still probably very important both because others don't have your script, and ideally, any work done on hledger-web should be useful to non-Sandstorm users if possible.

[simonmichael]

Editing the full journal was supported previously. You could “import” by just pasting in the whole new journal. The code is probably just commented out.

I disabled to make hledger-web safer/easier to deploy out of the box, in the scenario where you want to expose your accounts to viewers or collaborators, without worrying that your journal could be rewritten or broken. I figured technical folks can still edit the file by other means, or reenable the edit form and set up an access controlling proxy if needed.

We should add some built in access control and then lots of rich editing features.

For export, I think there is a route that will serve text right now - /raw or something.

A related general question is: does it make sense to keep building on/evolving the current web app, which is server side and in places ad hoc ? Or would a new start or new architecture/ framework be better ?

[ocdtrekkie]

@simonmichael The plan for what to do with hledger-web should likely be closely aligned with how much interest there is in working on it. (In my case, I am not a Ledger user, I'm just hoping to see what we can do to hit that "satisfying" mark for the Sandstorm version.) I don't know any Haskell, so the amount I can contribute is minimal as it is.

From the changelog, it appears the raw view, edit option, etc. was all dropped back in 2014 from the UI, my guess is there's a good chance it's significantly broken (even if I could find it).

[simonmichael]

What is the expected workflow of an average hledger-web user?

A little more context: an early goal was: "What's the quickest cheapest maintainable way to make hledger usable by less-technical folk ? How can I let my non-CLI-using spouse see/add data ?" The current app is strongly influenced by two forces: 1. scarce developer/maintainer time and 2. desire to be usable by non-techies.

Of course, it can grow in any direction we want.

[zarybnicky]

As a hledger-web (but not Sandstorm) user, the most painful issue for me is #586.

However, if hledger-web was to become a full Sandstorm app (or a self-hosted solution), I think at least splitting the codebase into a well-defined API and a client should help a lot. I have no idea how much work extending the current Yesod app would be (e.g. adding proper AJAX authentication to solve the above import/export vs access problems), but one thing I thought about a bit last summer was a mobile (h)ledger app, and this would help it out quite a bit.

[ocdtrekkie]

For quick reference, since it took a while to find it, https://github.com/simonmichael/hledger/commit/ec51d28839b2910eea360b1b8c72904b51cf7821 is the removal of all of the old features hledger-web had. JournalEditR.hs and JournalEntriesR.hs have been completely deleted since then, which I gather are required files for those routes.

@zarybnicky : Fun bounty fact: Bountysource doesn't take their cut until you withdraw a bounty, so you could also use your bounty from #430 to reassign it to your pet peeve of choice. ;)

[zarybnicky]

@ocdtrekkie Nice. I guess I might as well dump it back into hledger, assuming my claim gets approved.

[simonmichael]

@ocdtrekkie agreed. Good to know more about your situation. It's kind of you to contribute to a project you don't use! (I will scheme on how we could tempt you. :)

This issue I opened was quite broad and vague. Defining some success/quality criteria or milestones for the sandstorm app would be quite useful. You probably have a lot of insight into what's expected in a good/well-regarded sandstorm app. Promoting/teaching about sandstorm in the hledger community is also pretty useful, most don't know about it and I have no real sense of how many sandstorm and sandstorm hledger users might exist, the health and growth rate of that community, etc.

I can't focus too much on growing the hledger-web app myself (except when I have a sudden personal need for it, which at the moment is rare). I feel sure we can attract more hledger-web devs though, especially if we can show people are using it. There's a bit of chicken and egg here.

@zarybnicky (weird; github doesn't autocomplete your nick), what do you mean by "a full Sandstorm app" ? If you feel like hacking a bit, we would love the help and I will mentor. Assuming sticking with the current codebase, next step would probably be to migrate it to latest yesod, then adding authentication should be a breeze. If exploring client-side alternatives, you could build it in to hledger-web, or use the more recent hledger-api if that's preferable.

[ocdtrekkie]

From a "solve issue 425" standpoint, I see two categories of issues with HLedger Web on Sandstorm.

1. People who hadn't heard of HLedger Web but found it on Sandstorm may have thought it was going to work like a traditional budgeting app and didn't know the right way to use it. I'd argue between the better description, and screenshots soon to come, this should be pretty much solved.

2. People who want to use HLedger (and know how to do so) specifically with their Sandstorm server don't have a good experience doing so. Which, since HLedger is primarily a CLI-based app, means we need to figure out how to make grabbing and working with your ledger less painful. So this is why I kinda lean to import/export as a top issue to resolve.

Adding editing back, permissions, etc. would be really good things, but will also require significant developer effort, which doesn't currently appear to be there. Whereas a smooth process for bringing a ledger in and out of your server means you can install/run hledger's CLI wherever you want, and store up your data in a central spot.

[simonmichael]

PS essentially, stopping the negative reviews (by turning them positive) is what this issue wants to accomplish.

[AaronM04]

I would add another reason for a suboptimal experience to category 2 above: scalability. hledger-web makes an HTML file containing all transactions and sends that. This becomes very cumbersome as the number of transactions increases. I believe this was mentioned on some other issue here. My workaround is a script to rotate the ledger file (this involves exporting and importing).

[zarybnicky]

From what #327 says, adding permissions is just a question of reading the X-Sandstorm-Permissions and conditionally displaying the relevant UI elements. Supporting that won't be difficult, just re-adding the old code and making sure it works.

That would solve the authentication problem on Sandstorm but not on standalone hledger-web. Though once we have this, adding proper authentication and hiding these features for certain roles/auth. levels can be a whole separate follow-up issue.

I'm not in the best position time-wise right now, but I think I'd be able to spare some time to do the first part, essentially solving #327 (I'd like to solve #586 as well, but that'll come later). Long-term though, I'd really like to see hledger-web split into a server and a client, as that would open several new avenues (hledger-cli querying a remote server, a mobile application, ...).

Edit: I missed that hledger-api exists. In that case, splitting hledger-web into a server and a client gets somewhat easier, though that'd also require significant changes to hledger-api (e.g. not only querying the file but also writing to it, authentication, etc.)

[simonmichael]

I was intending to do built-in access control first, useful for all users, with sandstorm integration as a thin layer on top. (Until I hear otherwise, I continue to think of sandstorm as a relatively niche part of our [potential] user base.)

Re architectural changes, feel free to start new prototypes at least to validate the idea (like the example in hledger-api); there's room for multiple different web tools.

[zarybnicky]

Yes, general access control would be good, but given the sandstorm headers, I thought it would maybe half the work to implement this. Once it is done, then this functionality could be exposed as a set of hledger-web flags (--view-only, --edit, ...). Further improvements, like login, will likely need extensions to the hledger file format, unless the extra info is to be stored in a separate storage.

[ocdtrekkie]

The existence of this option for Kanboard suggests that using a HTTP header to authenticate the user is not an unheard of strategy outside of Sandstorm: https://docs.kanboard.org/en/latest/admin_guide/reverse_proxy_authentication.html

Of course, that solely grabs the username from a HTTP header, whereas Sandstorm supports roles. But I would assume that perhaps there is some way that you could configure a reverse proxy, based on what user has authenticated, to offer up an HTTP header listing that user's roles.

Presumably then, one might offer up X-User-Permissions or something as a default header for setting roles, but make it configurable, so that X-Sandstorm-Permissions can be used instead.

[simonmichael]

Whatever you think best, as long as the overall architecture does not become too sandstorm-specific.

[ocdtrekkie]

Yeah, I think the best case is for the hooks to be there to control permissions in a few different ways, and then the options are there to use it for Sandstorm or elsewhere.

The hledger-web flags, for example like --view-only, --edit, mentioned above, wouldn't help with say, different users having different permissions, but would allow us to extend hledger-web's featureset without surprising or upsetting users: For example, I would argue the default behavior without a permission-related flag, would be for hledger-web to assume --add-only. So that hledger-web started with the same command that is currently used today, would not change what it can do. But a user who wanted to add edit capability to theirs could do so just by editing the command.

[ocdtrekkie]

I've added #811 to complete bullet one of my defined issues here: https://github.com/simonmichael/hledger/issues/425#issuecomment-395230454

[simonmichael]

Checking in five years later - https://sandstorm.io is still a thing, and has had checkins within the last year. But we don't seem to have anyone motivated to enhance hledger for it. If that changes, we can reopen this (or a new issue).

I think app pages (ours: https://apps.sandstorm.io/app/8x12h6p0x0nrzk73hfq6zh2jxtgyzzcty7qsatkg7jfg2mzw5n90) no longer show user comments, so that part is no longer a problem.

[ocdtrekkie]

Yeah I think at some point I would love to go in and refresh some packages like this, I know you did some great work on the web UI after my last release, but since nobody is currently bugging me about the app it is hard to justify putting it on top of my long pile of stuff I should do.

We have some cool stuff in the works this year which should bring up some new interest in the platform, and hopefully that will lead to some package update work as well. :)

[simonmichael]

Great! Sandstorm seems cool and I'd be happy to see this come alive again.

[ocdtrekkie]

I'm using Hacktoberfest to push me to do some long-procrastinated contributions, no guarantees, but I've kicked open https://github.com/ocdtrekkie/hledger/tree/update-sandstorm to see how far from success we are. Right now it just updates the standard Sandstorm packaging bits to modern Debian.

[ocdtrekkie]

Okay, so the current issue I'm hitting:

/opt/app/.stack.yaml.lock1068-0.tmp: withBinaryFile: resource busy (Text file busy)

It happens when running stack setup.

Research suggests others have noticed it happens when running out of a Virtualbox shared folder, of which vagrant-spk relies on as well. Actually, now I'm reading it might only happen when your Virtualbox is on Windows as well. So if that's the case, I should be able to proceed later tonight on my Linux box (or even via WSL2, which I haven't tried vagrant-spk over... yet.)

[simonmichael]

Try removing that file ?

[ocdtrekkie]

Guessing since you replied by email, you didn't see all my edits to the above comment. I think I sourced the problem as "attempting to use Windows". Which by now I should really learn not to, since I had a different bug I also traced out to "attempting to use Windows" already today. :D

[simonmichael]

Correct! I did not.

If you can reproduce it reliably, maybe it's something for the stack bug tracker. Also note there are two ghc install methods available these days, using stack or using ghcup (https://www.haskell.org/ghcup/guide/#stack-integration)

[ocdtrekkie]

Okay, after fighting with Virtualbox for a bit, and a lengthy build process... I am at:

hledger-web: getXdgDirectory:getHomeDirectory:getUserEntryForID: does not exist (No such file or directory)

[simonmichael]

Feel free to ping me on the hledger chat, and we could troubleshoot things interactively. Otheriwse, I'm guessing you're running in some kind of Debian VM on Windows ? What does env show (excluding anything private) ?

[ocdtrekkie]

This would be running HLedger Web in a Sandstorm sandbox on Debian Bookworm (the Vagrant setup), on a host which is Debian Bullseye, I believe. I gave up on Windows for this project.

Today's a but busy for me, I won't be able to test anything until tonight.

[simonmichael]

No problem, chat me any time if I can help troubleshoot.

[ocdtrekkie]

It looks like getXdgDirectory is trying to locate the writeable user profile folder to store data in. That shouldn't work in a Sandstorm sandbox, and never should have. The only writeable mount in Sandstorm is /var. Has there been any changes to HLedger which would explain why it's looking for that now, and is there a configuration to set the storage directory more explicitly?

https://hackage.haskell.org/package/directory-1.3.8.1/docs/System-Directory.html#g:3

[ocdtrekkie]

I'm guessing from https://hledger.org/1.31/hledger-web.html#options that maybe if I try to pass --file, I can avoid it trying to identify the home directory, but I'll need to go figure out where the existing package stores the file to configure it correctly.

[ocdtrekkie]

Looks from a test of the existing app that it's /var/lib/hledger/Ledger. So tonight, I'll try to pass that into the --file config option and see if that gets it moving.

[ocdtrekkie]

Guess we already do that: https://github.com/simonmichael/hledger/blob/master/.sandstorm/launcher.sh#L32

Is there any formatting nuance that would cause a problem? I noticed that the docs both show options specified with and without an = sign.

[ocdtrekkie]

Okay, you know what, I defined a HOME environment variable and that made it happy enough. It looks like the new version of the backup wants to store a .cache somewhere, and that's what pitches a fit about not having a home folder to use. It looks at a cursory glance that all of the permissions data is already in the manifest, so I'm going to go ahead and kick open a PR.

I gotta go find the app signing key, and then I'll post a test grain with sharing links with each permission level for testing, since my understanding of plain text accounting is still somewhat limited.

[simonmichael]

This has been a long issue discussion spanning seven years. Sandstorm no longer shows reviews, so the original issue is effectively resolved (and we can't recall exactly what people were complaining about). But with many updates done on both the hledger and sandstorm sides, I believe most pressing issues were fixed and once #2102 lands I guess the fixes will be delivered to sandstorm users.

Thanks to both @zarybnicky and @ocdtrekkie for your hard work. I can't tell if the original (tiny) bounty was paid - if you don't object let's round it to $100 for each of you - same procedure as https://hledger.org/regressions - and then I'll close this issue.

[ocdtrekkie]

I was not really bounty-seeking for this project (and I know Bountysource is deader than dead), but I suppose I would be remiss if I didn't mention we are going to try to solicit funding for some upgrades to Sandstorm soon, and collective-to-collective contributions are fee-free: https://opencollective.com/sandstormcommunity

[simonmichael]

That's interesting, good to know. I am happy to pay the bounty ($100 for each of you) to recognise sustained thankless work, and motivate others; or to transfer it to the sandstormcommunity collective as you prefer.

[simonmichael]

The sandstorm hledger-web app has been updated to the current hledger-web release, 1.31, and there's a new shortcut url to get there (https://sandstorm.hledger.org). @ocdtrekkie 's bounty was contributed to the Sandstorm collective. @zarybnicky, claim your bounty at your convenience. Thank you all!