Unclear how to go from github -> operating installation

[gwhitney]

I did read on the LedgerSMB website that "probably the fastest way to get a development environment set up is to use the Docker development image". But despite trying to follow all of the Docker installation instructions for LedgerSMB, I was repeatedly unable to get a working Docker-based installation on my Debian 9 Linode instance running. On the other hand, getting an installation working from the release tarballs went reasonably smoothly. So I would like to get running from the github source. I elaborate below -- thanks for considering whether there are any pointers you can give that would help in this situation. I appreciate it.

Background: I am interested in trying to make LedgerSMB more plausible for use to keep the books for a not-for-profit that I work for. (For example, I absolutely need to be able to mark assets with a "Fund" business unit, not just expenses and revenue, and do a balance sheet by Fund, rather than just revenue/expense statements by Fund -- that doesn't seem to be possible in current LedgerSMB.) So I need to get a system based on the github master working.

What I tried: I cloned the repository, and pointed my apache2 reverse proxy that was working with the tarball install to the clone, and ran starman from the clone, and of course, it responds but just with "If you are seeing this, there is probably something wrong with your dojo setup." So after searching around for a while, I found the README.md in UI/js-src. So I did the git submodule init and the git submodule update as advised, and I had to npm install uglify-js (but not sure what working directory I should ideally have been in to do that) and then tried the recommended command from that README.md, namely ../util/buildscripts/build.sh --profile lsmb.profile.js from the UI/js-src/lsmb directory. But now that script produces 13 warnings and 1197 (!) errors; I list examples of each type I observe below. And unfortunately, not being at all familiar with Dojo and only moderately familiar with Javascript, definitely not an expert, I am fairly stumped.

Is there a clear description anywhere of how to go from github master to a working installation? Obviously I need to be able to do this to have any chance of contributing to LedgerSMB. Thank you very much in advance for any pointers/information/thoughts on this. I will be very grateful.

Example warnings and errors from UI/js-src/util/buildscripts/build.sh:

warn(216) dojo/has plugin resource could not be resolved during build-time. plugin resource id: dom-addeventlistener?:./aspect; reference module id: dojo/on

warn(224) A plugin dependency was encountered but there was no build-time plugin resolver. module: dojo/request; plugin: dojo/request/default

error(307) Failed to evaluate module tagged as pure AMD (fell back to processing with regular expressions). module: lsmb/node_modules/commander/index; error: TypeError: Cannot read property 'EventEmitter' of undefined

error(356) The optimizer threw an exception; the module probably contains syntax errors. module identifier: dojo/AdapterRegistry; exception: TypeError: UglifyJS.Compressor is not a function TypeError: UglifyJS.Compressor is not a function at uglify (/home/ledgersmb/LedgerSMB/UI/js-src/util/build/transforms/optimizer/uglify_worker.js:18:30) at process. (/home/ledgersmb/LedgerSMB/UI/js-src/util/build/transforms/optimizer/uglify_worker.js:61:17) at process.emit (events.js:315:20) at emit (internal/child_process.js:906:12) at processTicksAndRejections (internal/process/task_queues.js:85:21)

[There are literally over a thousand errors and warnings in all, but as far as I can see by browsing through them, they are all similar to one of the above types, with the "error(356)" type the most common by far; as far as I can see, the TypeError in all of these is UglifyJS.Compressor is not a function.]

[ehuelsmann]

hi @gwhitney thanks for taking the time to file this issue!

As for your Docker experience:

I did read on the LedgerSMB website that "probably the fastest way to get a development environment set up is to use the Docker development image". But despite trying to follow all of the Docker installation instructions for LedgerSMB, I was repeatedly unable to get a working Docker-based installation on my Debian 9 Linode instance running.

too bad you're running into these problems. Maybe you can mail devel@lists.ledgersmb.org with the problems you're running into so we can improve the experience for the next person trying the same.

As for your problems preparing an environment for development -- even if you can't install Docker, the Dockerfile (Makefile for docker) provides a nice recipe of commands to be executed to build an environment. For the master branch, you can find the commands here: https://github.com/ledgersmb/ledgersmb-docker/blob/master/Dockerfile

If I would build an environment today (based on Debian Buster), I'd run these commands:

DEBIAN_FRONTEND="noninteractive" apt-mark hold sensible-utils && \
  DEBIAN_FRONTEND="noninteractive" apt-get -y update && \
  DEBIAN_FRONTEND="noninteractive" apt-get -y upgrade && \
  DEBIAN_FRONTEND="noninteractive" apt-get -y install \
    wget ca-certificates gnupg \
    libcgi-emulate-psgi-perl libcgi-simple-perl libconfig-inifiles-perl \
    libdbd-pg-perl libdbi-perl libdata-uuid-perl libdatetime-perl \
    libdatetime-format-strptime-perl libio-stringy-perl \
    libjson-xs-perl libcpanel-json-xs-perl liblist-moreutils-perl \
    liblocale-maketext-perl liblocale-maketext-lexicon-perl \
    liblog-log4perl-perl libmime-lite-perl libmime-types-perl \
    libmath-bigint-gmp-perl libmodule-runtime-perl libmoose-perl \
    libmoosex-nonmoose-perl libnumber-format-perl \
    libpgobject-perl libpgobject-simple-perl libpgobject-simple-role-perl \
    libpgobject-type-bigfloat-perl libpgobject-type-datetime-perl \
    libpgobject-type-bytestring-perl libpgobject-util-dbmethod-perl \
    libpgobject-util-dbadmin-perl libplack-perl libfile-find-rule-perl \
    libplack-middleware-reverseproxy-perl \
    libtemplate-perl libtext-csv-perl libtext-csv-xs-perl \
    libtext-markdown-perl libxml-simple-perl \
    libnamespace-autoclean-perl \
    libimage-size-perl \
    libtemplate-plugin-latex-perl libtex-encode-perl \
    libclass-c3-xs-perl \
    texlive-latex-recommended \
    texlive-xetex fonts-liberation \
    starman \
    libopenoffice-oodoc-perl \
    ssmtp \
    lsb-release && \
  echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list && \
  (wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -) && \
  DEBIAN_FRONTEND="noninteractive" apt-get -y update && \
  DEBIAN_FRONTEND="noninteractive" apt-get -y install postgresql-client && \
  DEBIAN_FRONTEND="noninteractive" apt-get -y autoremove

export LSMB_VERSION=1.7.0-beta1
export NODE_PATH=/usr/local/lib/node_modules

wget --quiet -O - https://deb.nodesource.com/setup_8.x | bash -
DEBIAN_FRONTEND="noninteractive" apt-get -y update && \
    DEBIAN_FRONTEND="noninteractive" apt-get -y install ${DOJO_Build_Deps} && \
    cd /srv && \
    git clone --recursive -b $LSMB_VERSION https://github.com/ledgersmb/LedgerSMB.git ledgersmb && \
    cd ledgersmb && \
    (curl -L https://cpanmin.us | perl - App::cpanminus) && \
    cpanm --quiet --notest \
      --with-feature=starman \
      --with-feature=latex-pdf-ps \
      --with-feature=openoffice \
      --installdeps .  && \
    npm install uglify-js@">=2.0 <3.0" && \
    make dojo

Did you run something similar? (Please note that I install Uglify lower than 3.0!)

[ehuelsmann]

Let us know if this helps!

[ehuelsmann]

Also, please note that this procedure installs into /srv/ledgersmb . You may need to remove the cd commands to install into the current directory

[ehuelsmann]

Maybe you can list the exact commands you executed here so we can help you identify differences with what we had expected. With the information you provided, the above is what I can do for you.

I'd appreciate if you would create an issue detailing your docker problems in the ledgersmb-dev-docker project.

[ehuelsmann]

Additionally, let me invite you to join us for chat at https://riot.im/app/#/room/#ledgersmb:matrix.org so we can help you more interactively to get up and running.

[gwhitney]

My apologies, other priorities took me away from this for a bit - as I mentioned, I am finance director for a not-for-profit, so I can only look into using LedgerSMB in my spare time. I will continue efforts on this whenever possible -- but right now, http://ledgersmb.org is not responding, some what stymie-ing my efforts to try the Docker installation again from the top as described, recording what happens from my point of view, as you requested above.

[gwhitney]

Have failed once again to get the docker version running. Will open a separate issue detailing that.

[ehuelsmann]

@gwhitney, Now that you succeeded to get the Docker version running, I'd like to help resolve this issue too, if from your perspective, it's still desirable. We're happy with any and all help we can get. However, that help doesn't need to be development effort. It could also be specification of BDD scenarios, writing of documentation, detailing user experience problems, etc.

[ehuelsmann]

Maybe https://ledgersmb.org/content/using-docker-develop-ledgersmb could be of help getting started with a development environment?

[ehuelsmann]

Also, you need to recursively checkout submodules. I've updated the docker development document referenced above to include that step.

[ehuelsmann]

Also, please feel free to comment on https://ledgersmb.org/community-guide/community-guide/development

[gwhitney]

Yes, I am actively pursuing getting the github master LedgerSMB up and running. I think I am close. I will be frank: Although the Docker method now works great for getting the production version up thnks to your clarifications (and revisions to the repositories), I am not pursuing a Docker development environment. I am trying to just check out the github master directly on my Linode instance and configure everything so that it runs. For me, with no familiarity with Docker, having to then somehow "enter" the Docker image (I think) and edit within it and then exit and run the Docker (if that's even right), and then how do I commit to my github fork from "within" the image?, it just adds a layer of complication and confusion to a development environment, on top of having not used Perl for 15+ years (I once used it very actively professionally) and being only moderately experienced with Javascript. I would rather have the code "directly" in the Linode instance and using the instance and interacting with git and github "normally". From my point of view, since I can create and destroy Linode (or Digital Ocean or AWS EC2 or ...) instances so easily and cheaply, the instance is the container, so why would I have another layer?

All that said, I do expect soon to have a sequence of steps for initiating github master on a fresh vanilla Debian 9 instance, thanks to your feedback and guidance. Where would you like me to report on them? On this issue, or as a comment on the page linked to in your most recent entry here?

[gwhitney]

Hmm, I am clearly shooting at a moving target. I see that a very recent commit in master eliminated dojo altogether, and that there are no submodules to init/update any longer. This change seems to obsolete some of the various READMEs in the LedgerSMB file tree, even as it stands in master (e.g., UI/js-src/README.md). I strongly suspect there is a Docker container somewhere that somehow encodes the latest setup for these changes, but I don't really know how to find it -- neither ledgersmb-docker nor ledgersmb-dev-docker repos on Github have been updated since the change to LedgerSMB removing the submodules. So maybe there's something on DockerHub or something like that? But this now seems like an odd bifurcation of the package -- most on Github, but some critical setup information elsewhere. I will look at the links above to see if they shed any light.

[ehuelsmann]

hi. yes, your timing is unfortunate in that respect.

The key here is not a Docker container, though, but installing npm (preferably version 12) and running make dojo.

I'm sorry about the invalidated documentation. We're working to clean up the docs (and the whole new slew of development dependencies pulled in by make dojo too).

[ehuelsmann]

(Unfortunate in the sense that the removal was merged to master only 4 hours ago.)

[ehuelsmann]

Oh. And the docker containers already run make dojo, which means that they don't need to be updated. Same applies to the procedure for generating tarball archives. I'm assuming that once you get your system set up, things will be a lot smoother sailing.

BTW, the last time we changed our javascript this drastically was over 4 years ago which is probably something like a century in JS-land. We aren't likely to change the structure this much in the immediate future, although we do have some plans to use a more modern library to eventually replace Dojo (Vue is a likely candidate; we're still debating the transition though).

[ehuelsmann]

One last remark for my day today: Could you open an issue for the functionality you think is missing -- which you were going to add/contribute to the project? Thanks !

[gwhitney]

OK, thanks to all of your very-much-appreciated guidance, I was able to get the github master version of LedgerSMB working at least to the point of the database setup screen looking normal, by the following list of steps, from a fresh vanilla Debian 9 "Stretch" instance logged in as a non-root sudo-enabled user id which I will call "debian" for convenience's sake. Further, for this example, I assume I have made mydomain.tld and www.mydomain.tld resolve to this instance.

A) Get the postgres server running and configured: 1) sudo apt-get install postgresql postgresql-server-dev-9.6 postgresql-client-9.6 2) sudo su - postgres -c 'createuser -S -d -r -l -P lsmb_dbadmin' [you will need to enter a password here that you must save for future use!] 3) sudo nano /etc/postgresql/9.6/main/pg_hba.conf [Edit the file to match what's given in the section "Configuring database access rights" on https://ledgersmb.org/index.php/content/installing-ledgersmb-17 ] 4) sudo systemctl restart postgresql

B) Install nodejs: 1) curl -sL https://deb.nodesource.com/setup_14.x | sudo bash - 2) sudo apt-get install -y nodejs

C) Grab the LedgerSMB code and its Perl dependencies: 1) sudo apt-get install git make gcc texlive-base texlive starman 2) git clone https://github.com/ledgersmb/LedgerSMB.git [or, of course, use the URL for your fork of LedgerSMB] 3) curl -L https://cpanmin.us | perl - --sudo App::cpanminus 4) sudo cpanm --notest --with-feature=starman --with-feature=latex-pdf-ps --with-feature=latex-pdf-images --with-feature=openoffice --installdeps ./LedgerSMB

D) Generate production Javascript from the sources: 1) cd LedgerSMB 2) make dojo

E) Get the starman service running: 1) sudo useradd -d /non-existent -r -U -c "LedgerSMB/Starman service system user" ledgersmb 2) sudo cp doc/conf/systemd/ledgersmb_starman.service /etc/systemd/system/ledgersmb-starman.service 3) sudo nano /etc/systemd/system/ledgersmb-starman.service Change WORKING_DIR to the LedgerSMB root, e.g. /home/debian/LedgerSMB in this scenario 4) sudo systemctl enable ledgersmb-starman 5) sudo systemctl start ledgersmb-starman

F) Get a webserver running to act as the https proxy for starman (this example uses apache, it could be nginx or other webservers): 1) sudo apt-get install apache2 certbot python-certbot-apache 2) sudo certbot certonly --apache -d mydoman.tld -d www.mydomain.tld

G) Finally, set up the proxy configuration for the webserver: 1) sudo cp doc/conf/apache-vhost.conf /etc/apache2/sites-available/ledgersmb.conf 2) sudo nano /etc/apache2/sites-available/ledgersmb.conf Replace the placeholders as directed; YOUR_SERVER_NAME is straightforward, SSL_CERT_FILE is /etc/letsencrypt/live/mydomain.tld/fullchain.pem, as is SSL_CHAIN_FILE, while SSL_KEY_FILE is /etc/letsencrypt/live/mydomain.tld/privkey.pem. Finally, WORKING_DIR is /home/debian/LedgerSMB 3) sudo a2ensite ledgersmb.conf 4) sudo a2enmod ssl rewrite proxy proxy_http 5) sudo systemctl restart apache

And now visiting https://mydomain.tld/setup.pl will/should let you configure your Postgres database for the LedgerSMB installation. In any case, looks good for me at this point.

A question remaining in my mind is exactly what I need to do when I modify the code in my checked out copy of LedgerSMB to test the changes. I presume if I change any Javascript in UI/js-src I will need to repeat the "make dojo" step; it seems to me that if I change Perl code I may need to do a "sudo systemctl restart ledgersmb-starman".

I will post an issue describing the needed changes for serious NFP accounting as soon as I have my database set up and can access the precisely relevant webpages.

Thanks again for all of the help!

[gwhitney]

Ugh, unfortunately the above installation is broken in some very subtle way. When I visit https://mydomain.tld/setup.pl, I get a reasonable looking database setup form. When I enter lsmb_dbadmin as the user, the password I created in step A2 above, and database name "test", and then hit the create button, I get back a page that just says

"Error! Unable to execute 'psql' at lib/LedgerSMB/Scripts/setup.pm line 237. dbversion: 1.9.0-dev, company:"

Trying to execute the same Perl code from the command line, I find that sure enough (admittedly as user "debian" not "ledgersmb"), if I execute:

perl -Iold/lib -Ilib -e 'use LedgerSMB::Scripts::setup; LedgerSMB::Scripts::setup::_sanity_checks();'

I get the same message "Unable to execute 'psql' at lib/LedgerSMB/Scripts/setup.pm line 237."

But on the other hand, if I execute

perl -Iold/lib -Ilib -e 'use PGObject::Util::DBAdmin; my $vh = PGObject::Util::DBAdmin->verify_helpers(); print "Hmm|$vh->{psql}|\n";'

(i.e., just trying to execute the body of _sanity_checks() from setup.pm directly), I get the output "Hmm|1|" -- so the psql helper can be executed! So with this installation I have created, something in LedgerSMB/Scripts/setup.pm is preventing the system call to psql --help (that verify_helpers tries) from executing properly.

It's very strange to me and I feel like it is getting into deep waters of Perl that I am no longer facile with. If any thoughts about what might possibly be wrong occur to anyone reading this, I would be extremely grateful for your advice.

[gwhitney]

Ah, false alarm. It turns out a Linode 1GB "Nanode" does not suffice to run LedgerSMB. You need at least a 2GB Linode instance. Resizing the instance has alleviated the problems in my last post.

[gwhitney]

Ugh, fits and starts. Once I resized the node, I was able to go through the database creation and first user creation without difficulty. But when I went to log in as the first user, it said the credentials were not valid. So I went back to setup.pl, and logged in as the dbadmin, did list users, and clicked on the page for my first user. Right at the top there is a box for resetting the password. But when I type a password into the entry box and click "Reset Password" I just get a page with the following message: "Error! Can't locate object method "save" via package "LedgerSMB::DBObject::User" at lib/LedgerSMB/Scripts/setup.pm line 1334. dbversion: 1.9.0-dev, company: "

I thought, oh, 2GB isn't enough either, but "free -m" shows 1250M used, 450M free, so that doesn't seem to be it. I will continue to investigate and post anything I find out, but if anyone reading this has any ideas I very much welcome them.

[gwhitney]

Hmm, I can't see any "save" method anywhere in the object hierarchy for LedgerSMB::DBObject::User either. So this actually seems like a bona fide bug. I will file a more conventional bug report as a separate issue from this. And then I will try creating a different database with its own different first user and see if I can get this github install to work.

[gwhitney]

OK, after checking in on issue #3746 as to whether I am even supposed to be able to reset a password from setup.pl currently, I went back to a fresh browser window pointed at setup.pl and created an entirely new database with a new user with full permissions. At the end of the process, I clicked on "Start using LedgerSMB" and again it took me to /login.pl, where I entered the username and the password (the company was pre-filled with the database name I just used). But as with the first try when I clicked on "Login" I just ended up at URL "https://connectionspace.org/erp.pl?action=root" with a page showing only the message "Credentials invalid or session expired". I.e., exactly the same as the first try. I don't think that I mistyped the user name and password twice in a row, but I will try a third database with username and password something VERY simple to be absolutely sure.

[gwhitney]

Unfortunately, exactly the same symptoms on a third try with super-simple user name and password. So there must be something subtly wrong with my installation. (sad face) And I really don't have much idea how to debug things at this level, since there must be a fair amount of state that's in place for a logged in session; I don't know how to recreate that in something I execute from the command line (like I could with the simple function calls in setup.pm). I am also not clear if there is a log file for the running LedgerSMB, and if there's a way to start up the starman service to make that logging more verbose? If so, then I could at least try to do some "insert print statement" debugging to see if I could get at why the authentication isn't working. But right now, even if I grepped the code for the error message I see and put some print statements there, I have no idea where that output would go... I am more than happy to poke around if someone can give me some pointers.

Absent some guidance, I am sufficiently stymied that the only way I see forward with my efforts to get our books online is to head back to Beancount/Fava and see if I can start implementing the necessary features there. Sadly, it's a significantly longer list than what's needed in LedgerSMB. On the other hand, Beancount/Fava is implemented in Python/Flask, so feels more comfortable and modern. But the biggest downside is that the actual transaction data in the underlying Beancount accounting engine is stored as a plain text file, as it was designed for personal use; I just don't think that's at all plausible for enterprise use. So a big first task would be to create a version of Beancount that uses a database for storing its transactions, to achieve some level of atomicity, security, and availability/throughput for the underlying data. I know that's all irrelevant to LedgerSMB, but you can see why I wish I could just get underway with adapting LedgerSMB to not-for-profit use.

[ehuelsmann]

Unfortunately, exactly the same symptoms on a third try with super-simple user name and password. So there must be something subtly wrong with my installation. (sad face) And I really don't have much idea how to debug things at this level, since there must be a fair amount of state that's in place for a logged in session; I don't know how to recreate that in something I execute from the command line (like I could with the simple function calls in setup.pm). I am also not clear if there is a log file for the running LedgerSMB, and if there's a way to start up the starman service to make that logging more verbose? If so, then I could at least try to do some "insert print statement" debugging to see if I could get at why the au

What starman invocation are you using? I'm assuming this is master?

[gwhitney]

Yes, this is master; and I am invoking starman via the supplied ledgersmb-starman service definition file, which has: ExecStart=/usr/bin/starman \ --listen localhost:5762 \ -I lib \ -I old/lib \ bin/ledgersmb-server.psgi

[ehuelsmann]

Ok. It's missing the now-required starman option --preload-app.

[ehuelsmann]

sorry about the terse responses. I'm just now waking up and getting behind a keyboard. The --preload-app function causes two things: a lot less memory pressure on your Linode instance (you may even be able to downscale back to 1GB), but the other is that all initialization code gets run before fork()ing the workers. I'm guessing you're running without configuration or without configuration which specifies an "cookie encryption secret". Not having such a config means that one gets generated for you. However, if the initialization code runs in each worker, each worker has its own secret, which in turn means that workers cannot validate each other's login cookie.

In other words: you're almost there.

[gwhitney]

Thank you very much, and I am impressed you were able to diagnose that remotely with limited information. I added the --preload-app to the service definition file and reloaded systemd and restarted the service, and indeed I can now log in to my test database. I presume you'll add the argument to a future iteration of doc/conf/ledgersmb_starman.service.

I am indeed running "without configuration." I will admit that I have not seen anything in any of the installation guides or instructions that talk about specifying a "configuration." If I missed something I should have done, I'd be very glad for further pointers. (You've been extremely patient with me!)

I think I am now in a position to actually start working on the enhancements necessary to use this to keep the books for the not-for-profit I'm working with. As requested, I will file a separate issue describing the first most important step that's needed.

I am going to go ahead and close this issue; if you think there's anything else I need to do for my installation to be really proper, or if you want it as a placeholder/reminder for adding the --preload-app flag to the boilerplate starman service file, I presume you can reopen it.

[gwhitney]

OK, my installation must actually be broken. First, there is #4754 that I just filed, which seems suspicious, but maybe really is a bug in master. But even if so, I have the following very basic problem: I go to Contact > Add Contacts and add a customer with name "Sally Fields" and Control Code "A-00004" and save it (so that it does then show up in Contact > Search if I select Entity Class "Customer" and then hit "search"). Then I go to AR > Sales Invoice. If I enter either "Sally Fields" or "A-00004" in the Customer entry box and hit "Update", I get an Error! popup with the message "Customer not on file! at old/lib/LedgerSMB/Form.pm line 343." I can't believe that there would be a bug in such a basic operation as entering an invoice, so there must be something wrong with my setup. Very sorry that I have not been adept enough at getting this all going, and sorry that I am just about at my wit's end.

[ehuelsmann]

Your setup isn't broken: our contacts UI isn't intuitive enough. You need to save an account before you get a customer or vendor. The account tab appears after you have the contact.

[neilt]

Since this issues was raised in 2020 we have significantly improved the instructions at https://github.com/ledgersmb/ledgersmb-dev-docker. If there are further problems getting docker development images running please open a new issue.

[neilt]

@gwhitney With regard to "I am interested in trying to make LedgerSMB more plausible for use to keep the books for a not-for-profit that I work for. (For example, I absolutely need to be able to mark assets with a "Fund" business unit, not just expenses and revenue, and do a balance sheet by Fund, rather than just revenue/expense statements by Fund -- that doesn't seem to be possible in current LedgerSMB.)"

There have been multiple LedgerSMB infrastructure improvements and we have more developers since 2020, both of which might make handling this easier today. If you are still interested please open a new issue.