search

blitzcode / hue-dashboard

A web interface for monitoring and controlling Philips Hue lights
http://www.blitzcode.net/haskell.shtml#hue-dashboard
279 stars 23 forks source link

App wouldn't build on Fedora 27 or macOS 10.13.4 #6

Closed jeanfrancoisgratton closed 6 years ago

jeanfrancoisgratton commented 6 years ago

Not sure if it's a Haskell framework (stack) or the app itself issue, but it fails in the few last steps of stack build with a missing dependency issue:

Configuring semigroupoids-5.0.1...
setup: At least the following dependencies are missing:
tagged >=0.8.5 && <1 && ==0.8.4

Same message on both platforms. This might be an outdated requirement, most likely, but I'm totally unfamiliar with the stack framework, or Haskell, for that matter.

How could I correct it, if it's an easy fix ?

blitzcode commented 6 years ago

Hmm. So that of course shouldn't happen, since stack has a fixed build plan with fixed versions and all packages on a release are mutually compatible and have all of their dependencies present.

I'm using an old LTS 5.15 (https://www.stackage.org/lts-5.15) from mid 2016 which includes tagged 0.8.4. So requesting tagged >=0.8.5 can't possibly work. Obviously semigroupoids 5.0.1 couldn't depend on tagged 0.8.5, otherwise it could've never been included into that snapshot.

But apparently it does. Now, looking at the old semigroupoids 5.0.1 in question, seems like somebody did a revision to it recently, years after the release:

https://hackage.haskell.org/package/semigroupoids-5.0.1/revisions/

Changing a dependency lower bound on a years old library included in Stackage LTS release seems like a very bad idea. No reason for the change is given. I filed a bug:

https://github.com/ekmett/semigroupoids/issues/71

Solving the immediate problem, there should just be a --ignore-version-bounds flag. AFAIK, there isn't. Probably another bug to file with the Stack people.

I could switch the code base to a newer snapshot, but then I'd also have to switch to a different compiler and who knows if that compiler version happens to have working ARM support. More bugs to file.

You could just edit the semigroupoids 5.0.1 cabal file in the .stack directory and change the tagged dependency back to its correct value ;-)

Let's see if we can't find a solution / workaround for this mess...

jeanfrancoisgratton commented 6 years ago

Ok, I'm sorry to say, but for the life of me Haskell and its build framework/toolchain eludes me. I cannot figure what to change. I did find a semigroupoids cabal file in $HOME/.stack/indices/Hackage/packages/semigroupoids/5.0.1 , but a quick grep of 0.8.[45] did not bring me anything. I kept a copy of the .tgz and modified entries left and right in the cabal file, tarred again the directory, same end result.

It looks like that irrespective of the changes I've made, that tarball keeps getting re-downloaded, somehow (.. I think ? as much as I understand that toolchain....)

I'm following your bug report on GH as well. We'll see what comes out of all this, I guess.

Your app looks splendid, really.

blitzcode commented 6 years ago

I don't blame you for being confused, Haskell's build system(s) can be a challenge on a good day ;-) Thanks for reporting it, though. This bug only shows when you have a clean slate, I wouldn't have noticed it myself. Too bad that your first contact with Haskell is such a disaster. Never seen this problem happen before!

Since I'm a bit grumpy about having to fix this mess, I'll use that energy to explain what happened.

Traditionally in Haskell you manually specified a version range for your dependencies. Like text >= 0.5 && text < 1.0. How do you know that range is correct? Most likely, you don't. Maybe you accidentally start using a feature from text 0.6 and your code won't actually work with 0.5 anymore. Maybe text 0.9 is released and introduces a breaking change. Maybe text 0.7.2 has some big bug in it. Also maybe you want to use library A & B relying on library C, but A wants C >=1.0 && < 2.0 and B wants C >= 2.0 and your build can't succeed. Or somebody releases a new version of a dependency with a bug that's still in your range and now your old code suddenly doesn't build anymore. IMHO, it's all a bad idea and in fierce competition with "monads are scary" for why people have given up on learning Haskell.

Fortunately, now there's Stackage. The idea here is they created a curated package set, basically a fixed list of packages with a single version each that all typecheck, build with each other & pass their tests. Instead of guessing version ranges you just pick a single snapshot and enjoy an immutable, semi-tested, always-building set of packages. And your stuff builds with equal dependencies on different machines, 3 years from now etc.

Now, because this version range stuff blows up in your face so often, sometimes time-delayed, people introduced what I'd consider a misfeature in Hackage, Haskell's package repository. It's called revisions. It allows you to go back and edit the dependencies of older package releases. Generally, this is done so that an old release that specifies text >= 0.1 is changed to text >= 0.1 && text < 0.3 when the 0.3 release of the text library breaks this old release. Here, they (IMHO) abused this feature pretty badly and completely changed the version range of a years old release to one that is impossible to satisfy by the Stackage snapshot I'm using. I can't imagine there's a convincing reason for this, neither would I expect them to go back and undo their change.

The Stackage people realized it's not acceptable for snapshots to be retroactively broken, they don't need this revision mess anyway, so now they're just ignoring them. Great! This unfortunately does not fix it for the old snapshots, like the one I'm using.

I did not really follow this entire mess in detail, so right now I'm a bit lost what the proper fix is.

Something you can do is add

- semigroupoids-5.0.1@rev:0

right below extra-deps: in stack.yaml

I'm not sure yet if that is what I want to commit as a fix, but it should allow you to proceed.

jeanfrancoisgratton commented 6 years ago

Your explanations of the Haskell environment are really enlightening and I thank you very much. Haskell always interested me, but I couldn't find a convenient use-case to justify diving into it. I dove in Ruby because of Puppet, dove into Python because of Ansible and general affinity with OpenStack. Problem is, you do not find many major ecosystems that broadcast "now with Haskell API" as far as I know.

Regarding the situation you've described, it's a mix of "unintended consequences", and "feature creep that went badly". Does LTS stand for what I think it stands, which is Long-term support ? They're defeating its meaning, for sure.

I'll try editing stack.yaml and will let you know about it. This way you could edit your README.md file with this hack if it proves to be good.

jeanfrancoisgratton commented 6 years ago

Your quick fix did the trick, as you expected. Your app runs perfectly now. A small glitch with the very first image as you can see at https://pasteboard.co/Hi63U9Y.png .. Image is missing, but if I click on the broken image picture and open it in another tab, it does show, it's the file /static/svg/bridge_v2.svg.

Not making a fuss of this :-)

A last question before I stop bothering you: how is stack doing its PATH research ? Right now if I try to move anything out of the git workspace where I've built the app, nothing will work.

My end goal here is to package everything (stack + libs/dependencies + hue-dashboard) in some format, be it tarball, RPM, .DEB, OSX-PKG, you name it. I do not want to go through what I've gone and want everything puppet/ansible-ready.

blitzcode commented 6 years ago

The LTS snapshots just mean that they're a set of snapshots of the Haskell ecosystem that will receive bug & security fixes for a few months without making you go to a new snapshot that might have feature & behavior changes. Each snapshot should be supported indefinitely in the sense that they describe an environment of compiler + libraries that stack should be able to recreate at any point in the future. This bug you stumbled across is basically an unfortunate interaction between a nasty but once required hack to support the 'version range' approach that intruded in unexpected ways into the more recently developed 'vetted, immutable snapshot' system. This is thankfully not going to happen anymore.

Good to hear at least the workaround allows you to build. I'll have to look into this a bit more to see if this is the fix I want to go with or if there's a better solution.

I got a Vagrant setup contributed to this project, maybe that's helpful for you. Otherwise the best general way to deploy Haskell apps (if you're not using Nix or building from source with stack) is likely to just ship a statically linked binary with everything included.

Looks like you found another bug! I haven't seen that one before, this should not happen. Since it's just a static image referenced in an tag, I can't really think of a good reason why it would be missing. What platform is the server / which browser? Did you try on a different OS / browser? Does this keep happening if you refresh?

jeanfrancoisgratton commented 6 years ago

Your application ran on a Fedora 27 updated with the latest packages (dnf upgrade just before launching your app). It ran in a Chrome (latest stable) browser running on MacOS 13.14.4.

I can spin up VMs for CentOS, Ubuntu, Debian, Fedora (not RH, yet) on my libvirtd setup at home, and see if the errors show up. Moreover, the laptop I'm writing this from is a Mac. I can thus cover much of what is needed here. I'll try to reproduce the bugs everywhere I can and provide you with logs, statuses, etc.

Could you tell me which logs, which files, you need ? I'll ship them once I get the results (sometime in the next hours).

I'll apply the stack.yaml "fix" everywhere

blitzcode commented 6 years ago

I've never once seen an issue were an image was not showing up on any host OS or client and don't have this problem with the current version which I've been running 24/7 on Mac/Linux. It's just a static image referenced by an img tag, so not a lot of fancy things that could go wrong.

hue-dashboard outputs a fairly detailed log, but I don't think there's anything particularly useful in there regarding a missing image. Never had an issue where some static resource was not being accessible by the browser, so there's not much logging related to that.

You can run stack exec hue-dashboard -- --trace-http to get some extra tracing from the HTTP server, maybe that'll be helpful.

I don't really have a good hunch why you're seeing this bug. For now the best I can suspect is that it's something odd only happening on a particular host OS / browser etc. ;-) Sorry that I don't have a more helpful suggestion, but for now I'd just be curious if you can reproduce this on another host / client combination or if the log says anything of interest.

jeanfrancoisgratton commented 6 years ago

Well, I've built the VMs, and your app is building on the 4 linux distros. Takes a bit of time, CPU steal is over the top as my VM images were a bit larger than needed, and are spread across 2 overworked hypervisors. It'll end up having been some browser glitch, likely, but we wont be able to say we haven't checked e-ve-ry-thing ! :)

Regarding something I did ask about earlier ..

Let's say, I've built everything under a non-root user, under /git/hue-dasboard/ . First attempt with stack exec hue-dashboard worked beautifully (you did write a great app).

Now, /git not being where I'd want my stuff to sit, let's say I moved the app somewhere else (say: /opt/local, or /usr/local)... 1) what would I need to move to get it to work, the install/lts-* directory is everything under it ? 2) what config change, and where do I do them ?

I understand that this question goes a bit beyond the scope of the original post in this thread, feel free to ignore :)

blitzcode commented 6 years ago

Search in hue-dashboard.cabal for the -dynamic flag. If you remove that you should get a large, statically linked executable and you don't have to distribute all those dependencies. Stack also has integration with Docker in case that's an option for you regarding deployment.

hue-dashboard just needs its config.yaml and the static directory in the working directory.

jeanfrancoisgratton commented 6 years ago

Noted for hue-dasboard.cabal, and you can close this issue while I go and punch myself in the face. I could not reproduce the missing svg issue on a different browser (still there on Chrome...until I cleared its cache). Really, I should have thought about clearing the cache !

Docker might or might not be the solution. I'm not yet sure I like docker, although I work with it day in/day out. I'll go with the statically linked executable.

Your app is what I needed, and I thank you for it, and your great support !

blitzcode commented 6 years ago

No fix is the easiest fix, great ;-)

I don't really know much about devops & Linux system administration. So far I've only deployed my Haskell applications through Stack/Cabal. If you find something for hue-dashboard that might be useful for others, let me know. I could update the docs or add a script or configuration file for one of the various container / automation tools etc.

blitzcode commented 6 years ago

Looks like adding allow-newer: true also fixes this issue, this time by actually allowing the different dependency from the revision to be used.

jeanfrancoisgratton commented 6 years ago

I'll await your commit and then I'll pull your repo and create ansible + puppet scripts to deploy the statically linked version of hue-dashboard, if you wish.

I've actually started creating a work environment to do so. Not decided yet about Ansible, really, but the puppet module will be a simple one (too easy to break things on existing setups, especially of different versions). Still, it'll be up to you to merge my branch if you find it useful :-)

jeanfrancoisgratton commented 6 years ago

Hi ! I'm almost done creating an ansible role to deploy hue-dasboard. I'm statically linking the executable, and I did try something I took everything from /git/.stack-work/install/x86_64-linux-tinfo6/lts-5.15/7.10.3 and copied it into /usr/local/hue/

When I then try /usr/local/bin/stack exec /usr/local/hue/bin/hue-dashboard it still looks like it'll rebuild the whole app :

15:20:38|jfgratton@tfed:~]: /usr/local/bin/stack /usr/local/hue/bin/hue-dashboard
stack: WARNING! Expecting stack options comment at line 1, column 1
stack: WARNING! Missing or unusable stack options specification
stack: WARNING! Using runghc without any additional stack options
gghc-tinfo6-8.2.2:   13.91 MiB / 122.86 MiB ( 11.32%) downloaded...^C

I get the impression that /var/ansible/hue-dashboard/git, where the repo was cloned and the app built is hardcoded somewhere but I cannot find where. Do not get concerned with the /var/ansible name; it's only where I've (manually) built the app to try to move it from, afterwards.

The app is linked without -dynamic in hue-dashboard.cabal, as you suggested.

Am I missing something ? I'm almost done to send you a PR for ansible :)

***EDIT : It looks like it only re-installed GHC (... where ??). Is there a way to avoid this ? My ansible playbook is completed, it'll leave the user the options to 1- change hue-dashboard's port, 2- link dynamically or statically, and 3- choose a target (install) directory once everything is completed.

blitzcode commented 6 years ago

Great ;-)

Don't launch the executable through stack outside of testing/development. Stack's job is building, it's not involved anymore once you deploy. Try running stack install hue-dashboard in the project directory and stack should copy the executable to a bin directory (and also tell you what it copied where). Then just use that binary, don't copy stack's build files. Don't forget to copy the static folder to where you put the binary.

jeanfrancoisgratton commented 6 years ago

Noted. Thanks for your crash course on stack and Haskell (by the way.. I'm starting to learn it, this week-end).

To make it clear, stack install hue-dashboard will choose which bin directory ? /usr/local/bin ?

Let's say I wanted it to install it somewhere else (say.. /opt/hue) which file would I need to modify, or which params would I need to use ?

jeanfrancoisgratton commented 6 years ago

Sorry, as you know, stack + haskell are not my thing (yet).. trying to do some automation on stuff you know nothing about is fun, though :)

blitzcode commented 6 years ago

It'll copy it to stack's local bin directory, see stack path. But just run it, it'll tell you what it copied where. It's not meant to be an actual install for your purposes, it's just the easiest way to get at the right executable without having to comb through the guts of of .stack-work/. This static executable should be self-contained and you can just deploy it anywhere.

Have fun with Haskell, it sure has a learning curve, but that's a huge part of the appeal. It's very different from any of the mainstream languages. Everything from the syntax to lazy evaluation, functional programming, purity, strong typing, abstractions like lenses and monads, its approach to parallelism, IO, parsing, data structures, managing state, error handling etc.

jeanfrancoisgratton commented 6 years ago

You do reply fast, considering our timezones (you're in DE, am I right ? I'm in CA)!

Noted for stack. I did enforce it with Ansible, one way or the other (there'll be a README.md file, not to worry !). Struggling right now with variable parsing on Debian; the whole playbook works just fine right now on Fedora/CentOS/RH .. Not bothering with MacOS, my dev setup is totally broken there.

I gather that you're not too familiar with Ansible, I'll leave my coordinates in the playbook if anything breaks so you won't have to bother with that.

blitzcode commented 6 years ago

Just 1AM in Europe, plenty of overlap ;-)

Yeah, if you could document your setup, that'd be very helpful.

jeanfrancoisgratton commented 6 years ago

Hi, stumbled upon something else. I need more than the hue-dashboard binary. I've managed to copy the executable at a place of my choosing, but it now complains about the static images . I guess I need to copy everything from static/* to my target directory, right ?

Is there something else I need to copy ?


ERROR | ThreadId 4   | 2018-04-30 15:14:32.200199
Can't load color picker image: static/color_picker.png: openBinaryFile: does not exist (No such file or directory)

INFO  | ThreadId 4   | 2018-04-30 15:14:32.200753 - Exiting, persisting configuration data...
INFO  | ThreadId 4   | 2018-04-30 15:14:32.201593 - Shutting down trace system
hue-dashboard: Can't load color picker image: static/color_picker.png: openBinaryFile: does not exist (No such file or directory)
[15:14:42|jfgratton@tcentos:~]: sudo updatedb
[15:14:48|jfgratton@tcentos:~]: locate color_picker.png
/var/ansible/hue-dashboard/git/static/color_picker.png
jeanfrancoisgratton commented 6 years ago

Ok, previous issue is fixed, but now I get into something else. Three things, actually. In my test setup, I've put the executable and the static directory in /opt/hue/

1) If I launch hue-dashboard from my $HOME directory like this: /opt/hue/hue-dashboard it will load $HOME/.stack/config.yaml, but won't be able to find the static graphics files (the error I've shown you above). 2) If I do this instead : cd /opt/hue; ./hue-dashboard , it'll complain that it cannot find the config.yaml file, but will continue. This time the binary finds the svg files, CSS, etc . 3) First time I successfully launch the application I need to push the link button on my Hue Bridge, that's OK. If I terminate it (CTRL+C), and then relaunch, it will ask again to press the Hue Bridge link button. I do not remember having seen this before, and it is not a workable solution for me, unfortunately.

You can see hue-dashboard's trace at https://gist.github.com/jeanfrancoisgratton/03f713492d3195f51daf2e62da2ebdb0

blitzcode commented 6 years ago

I mentioned before you need the static directory ;-)

The application will try to load the configuration from the current directory, that's also where it tries to find the static directory. This is just how things would be setup if you launched it from stack. The working directory would be the project root.

hue-dashboard will save its configuration and things like the username it registered with the bridge in the config file. In your trace it seems the configuration file is empty or contains invalid data. Did you check the file contents? Did you change the working directory between runs? (then it can't find its old file and just creates a new one, obviously needing to register with the bridge again). hue-dashboard will try to save its configuration on exit, but it might not be able to. Perhaps there's a system crash or OS specific behavior might kill the application before it has the chance to do so. It also writes its configuration every 15min. I run hue-dashboard for months at a time without a restart, I'd risk losing things like scenes or schedules if the host had a loss of power etc.

jeanfrancoisgratton commented 6 years ago

A last thing... I cannot push my local branch to your repo; I'm a Perforce guy, not a Git guy, the issue is most likely on my side.. I think ?

[15:57:29|jfgratton@laptopjf:ansible]: git push
remote: Permission to blitzcode/hue-dashboard.git denied to jeanfrancoisgratton.
fatal: unable to access 'https://github.com/blitzcode/hue-dashboard.git/': The requested URL returned error: 403
jeanfrancoisgratton commented 6 years ago

Sorry, you're right about the static directory.. apologies .. :)

I'll track what's happening with config.yaml. Since I will write a wrapper script to cd into my target dir and launch it from there, I think I might as well copy config.yaml there as well.

blitzcode commented 6 years ago

Nothing wrong per se, but you can't push into my repository. Fork my repository, create a topic branch, make your changes, send a pull request. I'll then do potential edits, rebase your branch and merge with my master. Your commits will then just show up on top of mine in this repository. That's a pretty common workflow for git.

jeanfrancoisgratton commented 6 years ago

I was thinking more of creating a remote branch on your repo, and pushing my local repo to that remote branch, then have a PR so you can merge against master. Fork is good too. I've pushed my commit and created https://github.com/blitzcode/hue-dashboard/pull/7

blitzcode commented 6 years ago

I had a look at the setup in the PR. A few things that would need a second look:

sudo curl -sSL https://get.haskellstack.org/ | sh
stack setup
git clone https://github.com/blitzcode/hue-dashboard.git
cd hue-dashboard
stack install --ghc-options '-optl-static' hue-dashboard
target_dir=/opt/local/whatever/bin/hue-dashboard
mkdir $target_dir
cp ~/.local/bin/hue-dashboard $target_dir
cp -r ./static $target_dir

Maybe there's a simpler solution?

jeanfrancoisgratton commented 6 years ago

Hi,

Admitedly, my shortcomings are more about me not knowing much about stack+haskell+hue-dashboard themselves (...and maybe not enough of RTFM) than anything. You did show a faster, simpler way to get things done than the way I chose.

About having two of your files copied and templated by Ansible, I agree it wasn't far from optimal, but I figured that since you do not plan to move from LTS' current version in the short term, and since I am tracking your repo, I could live with that; it does introduce a potential breakage in the whole ansible playbooks (...if you used -e variables while running the playbook).

Since your solution is a lot simpler (which I wasn't aware of), this PR is therefore useless, and I'm not vain enough to fight this losing battle :)

One thing I've lost sight of is that my 2 VPS and collection of VMs at home are all setup through puppet, so the underlying Ansible/Puppet infra is always present at home, whenever I spin up a new VM. This is not the case of any user who might clone hue-dashboard.git, so yes, it might be a bit of work (not much) to set things up.

... you did give me an extra incentive to get on Haskell ASAP, though :)

I guess you can close/reject my PR. You did bring up a way easier way to get things done than I did.

blitzcode commented 6 years ago

Will do, much success with your Haskell endeavors! ;-)