Roundup build: package will not be available in Debian apt for Stretch

[update by Jeremy] - my suggested path forward with Roundup appliance for v15.0 is noted in this comment.

Just came across this post regarding Roundup missing from the Stretch repository:

... "During this work a security issue came along and this made me realise that the architecture of roundup isn't exactly compatible with what I would expect from a proper Debain package."

"My advice: File a removal" ...

https://lists.debian.org/debian-python/2016/03/msg00129.html

FWIW

Hi @bormanst. Thanks for the link on this. I was aware that it was no longer in the Debian repos, although I hadn't researched exactly why. So you've saved me a little time! :+1:

My reading of your link is that the primary issue was due to licensing:

[...] [includes] some vendored code (javascript libs and fonts), 5 different licenses, and in 1.5.0 there is an offending file that has an incompatible licensing [...]

And the secondary issue was how the package could be kept secure (in the context of the way a Debian package should work. I.e. install to /usr/lib and/or /usr/share with end user adding customisation in /var/lib (or similar):

Roundup is not designed to be customizable in the way Request Tracker (request-tracker4) is. The latter supports local customizations (in /usr/local) and plugins either from source, Debian packaged extensions; without having to rewrite any files from the package in /usr/share.

[...]

Roundup is intended to be copied and heavily customized, although I can see a very small niche where roundup would contain only an instantiated demo tracker in /usr/share (I.e. no customizable config/authorization).

So my reading of that is, not that Roundup is insecure as such. The issue is that providing security updates via normal Debian package security updates would either break the users install (if they edited files in /usr/share) or not be applied at all (if they moved the whole thing somewhere else).

The former would be a pain, but the later could be quite dangerous to end users who may not realise their code was insecure (if they checked, they would have the security update installed - but they wouldn't be applied to the copied code).

So whilst not ideal for a TurnKey appliance, I'm pretty sure that they aren't referring to the core security of Roundup itself, just that it's design and architecture are not really compatible with the way that Debian does packaging and security updates.

Having said that, I'm not clear on how things would go if Roundup were installed via pip (as upstream advises), then heavily customised. When updated to a newer version, that may also break it? A further issue with a TurnKey install is that by default TKLBAM doesn't include pip installed packages, so any modifications made by the end user would not be included in the backup (unless again it were copied somewhere; thus reintroducing the other issue noted by Debian devs).

I see that Round is still hosted on SourceForge, although as I noted above, the recommended way to install it, is now via pip.

One way we could go, is we could could install it (via pip?) somewhere using a python virtualenv. Then we could include it as part of the backup. I'm still not sure how updating it would go though...

Perhaps it's a "too hard" problem for now and we should just consider leaving the Roundup appliance out of the library for v15.0? I'd appreciate any additional insight or thoughts you may have...

The beauty of your appliances is their minimalistic "fire and forget" nature. If the admin has to perform one-off security maintenance for a package, then they should understand how the package is installed and what else may be affected. It would be wise to have them install it themselves, if it's not too complicated, as most useful packages have adequate install info.

I haven't run through the gauntlet of your offerings, but packages (not appliances) like Roundup and SimpleInvoices, that won't get automatically updated with apt repositories, would probably best handled with a post-LAMP install script, i.e. TKL-LAMP with a TKL-LAMP-application selector to automate installs with your existing shell/python/overlay scripts. Then the admin can choose to automate or manually install the package (point them to your excellent scripts.) Either way, they will be made aware of what needs minding. Obvious exceptions would be packages that have their own self-update mechanisms like Drupal, etc.

My guess is that, if one is choosing Roundup for it's configurability/customization, then it's really not a stretch to assume they can handle TKL-LAMP and a manual install. So I would forego developing a such a specialized appliance unless it is in high demand. If it was a super tricky install, a massive download/gcc compile, or some kind of dependency hell, then that would be a different story about balancing the trade-offs.

Quality vs quantity is the age old question. One of TKL's best qualities is security so...

TKL-Roundup >> Roundup exclusive breach >> admin loses trust with TKL appliances.

TKL-LAMP + Roundup automation script >> admin responsible for Roundup install with notice >> Roundup exclusive breach >> admin aware of responsibility >> no loss of trust with TKL appliances.

The emailed security alerts option eases some of that concern, but not everyone selects that option.

And then there's the "...then heavily customised. When updated to a newer version, that may also break it..." issue. From my experience, most admins understand and handle security frustrations in stride on a daily basis. Those same admins, however, will do anything and everything to get rid of something that randomly breaks on it's own, especially if restoring a back-up can make it worse. That should be a non-starter which leaves one-off security maintenance as the only viable option, i.e. their problem.

All that being said, I believe Roundup to be a lingering application as opposed to something like Bugzilla which is used by the kernel community and won't be giving up the ghost any time soon. I'm not trying to make feature comparisons or cast shade on Roundup (it's been good to me) but I don't see it getting refactored to meet standards - ever. I would just end-of-life the appliance at v14.2 and pick it up again if it's status changes or you have to develop the pip solution for another appliance build.

FWIW

A more constructive response:

...One way we could go, is we could could install it (via pip?) somewhere using a python virtualenv...

I would concur with that approach; python virtualenv is very similar to composer and is able to "snapshot" the foundation of a "working" environment without worrying about the application itself. The major attractive feature of this method is that restoring from a back-up would pretty much guarantee a working copy minus any subsequent modifications. And since the current broken system would contain all the subsequent modifications, it would be a relatively straight forward and quick process to mitigate losses. Python virtualenv also allows for installing/re-installing the application to the same state which is never a bad feature.

While the "updating issue" may be of concern, updates break things all the time (recent kernel update broke my VirtualBox); admins just want to know that they can get out of the bad mojo as quick as they got into it. Your prescribed solution would provide that reliable option with the only predictable loss being the time it takes to merge subsequent modifications into the restored package.

Your analysis skill are excellent ;)

@bormanst - thanks for your feedback and your kind words. You raise some great points in your first response. Roundup does not appear to be under any serious development, so we may perhaps consider dropping it in the future. But TBH in the short term at least, my inclination would be to still provide a Roundup appliance.

On further consideration, I think rather than a full virtualenv, what may be a better option is to create a limited roundup Linux user account and install via pip into that. I.e. su as roundup user and install with pip using the --user switch.

That way we could still provide dependencies via Debian packages (so at least the dependencies would still get auto security updates). It would also be easy to include (a possibly user modified) Roundup in the TKLBAM backup. At least then if an update broke things, restoring to a known good state would be fairly straight forward...

Thanks again for your input and feedback. It's really valuable! :smile:

FYI, my assertion of a lingering application appears to be premature as it is the tracker used for the Python programming language itself; makes sense as it's written in Python. It's still on Python 2.x and I'm guessing that they'll do the upgrade to 3.x, whenever they deem it necessary, vs moving to a new platform. They're pip people so it would be fair to assume that pip will remain the preferred install method going forward. This would reinforce your proposed user switch solution as a wise choice; and also suggest that it should be robust vs a temporary one-off job.

https://en.wikipedia.org/wiki/Roundup_(issue_tracker)

It is being maintained as well:

http://hg.code.sf.net/p/roundup/code

FWIW

Thanks for the update and additional info @bormanst. Your findings back my initial research, so that's reassuring! :smile:

Another FYI time-saver: I completed a build (Google-GCE) using the "pip --user" solution that you suggested above and it appears to work just fine but I haven't had a chance to fully test it (hopefully by next week though.) There's a couple of Stretch gotchyas that are worth pointing out to save you some hassle:

Stretch uses mariadb as a drop-in replacement for mysql even though mysql-server is requested by apt-get. The drop-in is v10.1.24 and doesn't behave like myslq as roundup complains and fails on "...max key length..." errors of the type described here: https://stackoverflow.com/questions/43379717/how-to-enable-large-index-in-mariadb-10 . This solution would require changing the table creation scripts in roundup - booooooooo! Not optimal by any means.

Mariadb v 10.1.32+ adds a Global var innodb_default_row_format to address the issue so you can change the default ROW_FORMAT=COMPACT to one of the acceptable options but you have to add the separate repo to the apt lists: https://mariadb.com/kb/en/library/xtradbinnodb-server-system-variables/#innodb_default_row_format

The real mysql-server works just fine but also requires the addition of a separate repo.

The solution that I opted for was to apt-get mariadb-server which is already in the Stretch repo and gives you v10.2+. It addressed the "...max key length..." error and defaults to ROW_FORMAT=DYNAMIC. You have to shove "plugin-load-add = auth_socket.so" >> /etc/mysql/mariadb.conf.d/50-server.cnf under the section [mariadb] to resolve a socket access issue.

As you may already know, mariadb recently "hard forked" and has already introduced functionality not compatible with mysql so...

The sqlite backend works just fine, as well, but I'm guessing that's not your preferred option either.

FWIW

Thanks as per always @bormanst :smile:

Stretch uses mariadb as a drop-in replacement for mysql even though mysql-server is requested by apt-get.

FWIW, in Stretch, the mysql-server package is a meta-package which depends on default-mysql-server (another meta-package) which in turn, depends on mariadb-server (the actual software package which contains MariaDB).

The drop-in is v10.1.24 and doesn't behave like myslq as roundup complains and fails on "...max key length..." errors of the type described here: https://stackoverflow.com/questions/43379717/how-to-enable-large-index-in-mariadb-10 . This solution would require changing the table creation scripts in roundup - booooooooo! Not optimal by any means.

We've hit that same issue in a number of other appliances, although admittedly I think we've only explicitly noted it in relation to Rails - see this fairly lengthy background post by me that sums up my research around the issue from late last year.

Basically, we've worked around it by adjusting the config and modifying the table data before adding it to the DB (as you hinted).

As a side note, I anticipate that this will be a massive pain when users try to migrate data to v15.0 which is really unfortunate! We should probably consider adding a tklbam-hook to implement that, probably discussion for another thread though... Alternatively, I did (for a moment) consider providing some sort of wrapper script around the mysql command to address that on the fly (when creating the tables). But I'm not really comfortable with that level of transparent hackery, as I think there is significant risk that something could go wrong and we'd likely encounter weird edge case bugs.

The real mysql-server works just fine but also requires the addition of a separate repo.

Are you sure about that?! My reading suggests that MariaDB inherited the issue from MySQL 5.6 (the version that MariaDB 10.1 is based from/compatible with). I.e. I have seen numerous reports that MySQL 5.6 has (had?) exactly the same issue (e.g. see the note in this SO answer to the question you linked to above). AFAIK it was resolved in MySQL 5.7/MariaDB 10.2 (although you note, MariaDB actually implemented that into a later release of v10.1(.32+) so not 100% sure when MySQL addressed it, perhaps they also addressed it in a later release of 5.6?!).

The solution that I opted for was to apt-get mariadb-server which is already in the Stretch repo and gives you v10.2+. It addressed the "...max key length..." error and defaults to ROW_FORMAT=DYNAMIC. You have to shove "plugin-load-add = auth_socket.so" >> /etc/mysql/mariadb.conf.d/50-server.cnf under the section [mariadb] to resolve a socket access issue.

That sounds great, except are you sure that it came from the default Debian repos? According to the Debian online package DB, Stretch contains MariaDB 10.1 (10.1.26) - as does Buster/testing (10.1.29) and Sid/unstable too (also 10.1.29), only Experimental contains anything significantly newer (10.3.0).

At a guess, I suspect that if you've been playing with third party repos (e.g. MariaDB) that perhaps you've inadvertently installed 10.2 direct from them? Perhaps double check where it's installed from with:

apt-cache policy mariadb-server

I'd be interested to hear more...

Doh!!! Right you are. I happened to put the the repo addition into two different scripts while monkeying around with the errors and only deleted it from one. Clean GCE vm pulls in "10.1.26-MariaDB-0+deb9u1 Debian 9.1" just as you indicated. Sorry about any false hope there ;(

It was definitely the latest mysql-5.7 from "http://repo.mysql.com/apt/debian/ stretch" that I was using so no experience with 5.6 and related issues; I hadn't done a roundup install for a while and just wanted to rule it out as the bug source.

...wrapper script around the mysql command to address that on the fly... Woe to those that port a "working" script to another similar build ;)

...massive pain when users try to migrate data... At least it's going from smaller to bigger; having to truncate or split is a true pain. Did a Y2K alumni database conversion from JCL/Mark-IV mainframe to IBM/AIX for a large university that involved mostly the latter. Anyone that knows enough to actually require ROW_FORMAT=COMPACT vs DYNAMIC, etc., has probably been dealing with indexing/BLOB issues and should have a grasp of how to properly port the data - IMO.

I wish I could offer a decent solution but I have to have something working by next week and will probably stick with the mariadb v10.2 solution (and now a disclaimer.) I do have to add that "square pegging a round hole" should have been optional to begin with, i.e. a Global var should have been available from the "get go" to change the default; Oracle <=> evil empire. Oh where did the spilled milk go...

I should read your postings for breakfast as it may extend the life of my skull; your excellent rails posting alone would have saved a couple bricks in the wall :) It would have been the first thing I clicked on if the "googleverse" had presented it, but I was keying on roundup so...

Thanks again for all that you do.

Whilst as a general rule, monkey patching isn't ideal, doing that with RoundUp (as we did with Rails) may be an option too? But unfortunately I don't know enough about python to even begin to offer a practical suggestion on doing that...

PS thanks for your kind words. In retrospect, I should have perhaps posted most of that Rails/MariaDB post in on our blog. Google would have had a much better chance of finding it then (although perhaps not specifically related to Roundup still...).

Bingo??? Just completed a build using what may be an acceptable method for all of your affected appliances. It's kind of going to the moon and back but here goes...

1) apt-get docker-ce from the docker repository 2) run docker image for mariadb 10.1.33 3) install application using the docker database 4) docker exec mysqldump into home >> ~/dumped_databse.sql 5) create the 10.1.26 database (and user if required) 6) restore ~/dumped_databse.sql into the 10.1.26 database 7) set the 3 required globals: SET GLOBAL innodb_file_format=Barracuda; SET GLOBAL innodb_file_per_table=ON; SET GLOBAL innodb_large_prefix=1; 8) run script to ALTER TABLE $TABLE ROW_FORMAT=DYNAMIC for all tables in the database.

I was guessing that a minor point update (10.1.26 -> 10.1.33) didn't actually affect the "architecture" so I gave it whirl and it appears to work just fine; didn't have to modify 10.1.33 at all, so it's defaults must have already resolved the pertinent issues. I'm new to mariadb so you may want to rattle someone's cage, that's in the know, before embarking on this one. There's a few things worth mentioning:

1) Even though the docker image is stopped and removed (it has a switch that will automatically remove it when it stops), it fires up instantly for re-use (the image appears to be cached somewhere), i.e. to create another tracker, etc. 2) All that is necessary is a small bash script that gets docker going and sets configs for the install script to use the docker db, executes the normal install script, updates the new application configs to point to the 10.1.26 db, and shuts down docker again. 3) Docker isn't actively used so security issues shouldn't be much of an issue. The apt list can be removed and maybe more "lockdown" stuff if required. Again, sorry as this is my first docker experience so maybe someone with a slicker hat can be of more help. It can always be purged if it's not needed again. Roundup will require it to add new trackers so... 4) Except for the extra configuring, the normal install scripts can be left "as is" and will "just work" when the Strech repository gets a mariadb-server sans the issues. 5) You don't have to worry about 10.1.26 updates. 6) Updates to the applications shouldn't affect anything either, unless the install configs are changed which would require modifying the docker bash script. The docker bash script is probably best kept in a separate location so as not to get whacked.

While this appears to be incredibly confusticated, it is a straightforward process that is essentially a "dirty" database restore. Since there will be little to no data in a newly created database, "dirty" may not mean much, if anything. This alleviates all of the concerns with monkeying around with someone else's install scripts that can result in missed or messed up issues.

Although I just cut/pasted my notes and manually bashed it (twice); I should have a working script in another day if you're interested. I will do a more thorough testing of Roundup as well in the coming days. I don't want to appear to be jumping the shark but I didn't want you wasting time on another solution if this one fits the bill; especially since extensively modifying someone else's logic/scripts is never a quick slam dunk.

FWIW

p.s. The "[mariadb] plugin-load-add = auth_socket.so" issue got me twice in two days - doh! ;)

Thanks @bormanst - as we've mostly worked around it for our other appliances so far, we probably won't bother using this for building v15.0 appliances.

However, having said that, it is really good to know! And may be worth considering for future releases if we have further issues.

if you do ending up "bashifying" your workaround, please do share it. I'm almost positive that it will be useful for TurnKey users trying to migrate data from v14.x appliances to v15.0! Whilst it does sound a bit convoluted, if it's turned into a bash script, then it certainly sounds like it would be easier than other workarounds.

...if you do ending up "bashifying" your workaround, please do share it...

Finished and tested the script and have included the pertinent part below; while this partial is designed for my Roundup install (somewhat equivalent to your conf.d files), it sounds like you're interested in the HACK section. Everything installs and appears to run fine but I have not completed a thorough testing of Roundup other than add/edit/delete issues.

Couple of notes/updates:

1) I was queasy about the suggestion to run a script to ALTER TABLE $TABLE ROW_FORMAT=DYNAMIC for all tables in the database after import; I preened that idea from the tubes. However, upon further inspection of the dump file, the uniform style lent itself to a simple sed regex. So the row format issue is addressed prior to importing which alleviates something (BLOBs, etc.) possibly getting whacked. Easy to change to another desired row format as well. It should be relatively safe for all compatible imports of populated in-use dbs, not just the freshly minted; the migration goodness that you seem to desire ;) I'm no dump master so the provided sed expression may have to be enhanced to account for some wacky exports??? 2) Docker cleanup at section # TODO: more un-docker here??? may need more security attention as needed/required on a use case basis. It's needed to add more roundup trackers, as indicated before, so I leave it disabled but installed (no purge.) 3) There's several docker sources but the one I'm using is a one liner and appears to use binaries; it doesn't use the time eating gcc-bloat and compile route that many of the others use. May or may not be a security concern??? 4) DOCKER_PORT should just be different that the mariadb port. CONTAINER name can be anything as well. 5) localhost and 127.0.0.1 are NOT equivalent for this one, so don't get hosed by a search/replace, etc. 6) The docker container gets it's own local IP, so if performing some other fancy pants maneuver with a db user account, remember that the user will show up as a remote user which may prevent db access, i.e. USER@CONTAINER_IP and not USER@LOCALHOST or USER@127.0.01, etc. 7) The hack within a hack time delay is needed to ensure the service is running; the container reports back "true" when active, but the actual service isn't fully fired up. There's bound to be a more exacting method but, again, I'm new to Docker so no slick hat on this one :) I'm doing GCE builds that minimally use hazwells, so the interval may need to be increased if using slower cpus???

If you have any questions, please feel free to ask. Hope this helps.

Script partial from here to end:


#!/bin/bash -e

PACKAGE=roundup

\#  log check
MESSSAGE="Check packages: ${PACKAGE}"
echo ${MESSSAGE} >> ${LOG}

\#  check packages
PKG_CHECK=${PACKAGE^^}
if [ -z "${!PKG_CHECK}" ]; then exit 0; fi

APP_DIR=/var/www/support
\#  check dir
if [ -d "${APP_DIR}" ]; then
    echo ""
    echo ""
    echo "WARNING! ${PACKAGE} already exists."
    echo "WARNING! ${PACKAGE} may have already been initialized by this script."
    echo "WARNING! Re-running this script may corrupt existing system."
    echo "WARNING! Please review script in applications/${PACKAGE} for further details."
    echo ""
    echo ""
    exit 0
fi

\#  dependencies check: apache2
DEPENDS=apache2
if ! (systemctl -q is-active ${DEPENDS}); then
    echo ""
    echo ""
    echo "WARNING! ${PACKAGE} depends on the ${DEPENDS} service."
    echo "WARNING! Please ensure that ${DEPENDS} is active."
    echo ""
    echo ""
    exit 0
fi

\#  dependencies check: mysql
DEPENDS=mysql
if ! (systemctl -q is-active ${DEPENDS}); then
    echo ""
    echo ""
    echo "WARNING! ${PACKAGE} depends on the ${DEPENDS} service."
    echo "WARNING! Please ensure that ${DEPENDS} is active."
    echo ""
    echo ""
    exit 0
fi

\#  log configure
MESSSAGE="Configuring ${PACKAGE}..."
echo ${MESSSAGE} >> ${LOG}

\#  setup database
DB_NAME=${PACKAGE}
DB_USER=${PACKAGE}

\#  add roundup user for pip install
if [ $(grep -c "^${DB_USER}:" /etc/passwd) -eq 0 ];  then
    useradd -U -m -s /bin/bash ${DB_USER}
fi

BATCH_EX="--batch --execute"
ROOT_CREDS="--user=root --password=${DB_PASS}"
MYSQL_ADMIN=/usr/bin/mysqladmin
MYSQL_DUMP=/usr/bin/mysqldump

\#  set required global vars
QUERY="SET GLOBAL innodb_file_format=Barracuda; SET GLOBAL innodb_file_per_table=ON; SET GLOBAL innodb_large_prefix=1;"
mysql ${ROOT_CREDS} ${BATCH_EX} "${QUERY}"

\#  create database
${MYSQL_ADMIN} ${ROOT_CREDS} create ${DB_NAME}

\#  add database user
QUERY="grant all privileges on ${DB_NAME}.* to ${DB_USER}@localhost identified by '${APP_PASS}'; flush privileges;"
mysql ${ROOT_CREDS} ${BATCH_EX} "${QUERY}"

\#  temp perms for install
chown ${DB_USER}:${DB_USER} /var/www

\#  install roundup to user home
sudo -H -u ${DB_USER} bash -c "pip install ${PACKAGE} --user"

\#  set home access perms
HOME=/home/roundup/.local
chmod -R 0775 ${HOME}
chmod -R 0774 ${HOME}/share

\#  add symlinks
LOCAL=/usr/local
TARGET=/bin/roundup-admin
ln -sf ${HOME}${TARGET} ${LOCAL}${TARGET}
TARGET=/bin/roundup-gettext
ln -sf ${HOME}${TARGET} ${LOCAL}${TARGET}
TARGET=/bin/roundup-mailgw
ln -sf ${HOME}${TARGET} ${LOCAL}${TARGET}
TARGET=/share/roundup
ln -sf ${HOME}${TARGET} ${LOCAL}${TARGET}
TARGET=/lib/python2.7
ln -sf ${HOME}${TARGET}/site-packages/roundup ${LOCAL}${TARGET}/dist-packages/roundup
ln -sf ${HOME}${TARGET}/site-packages/roundup-1.5.1.dist-info ${LOCAL}${TARGET}/dist-packages/roundup-1.5.1.dist-info

\#  tracker creation
BACKEND=mysql
TEMPLATE=classic
OPTIONS="admin_email=admin,dispatcher_email=admin,tracker_web=/,mail_domain=${APP_HOSTNAME},mail_host=localhost,rdbms_password=${DB_PASS}"
sudo -H -u ${DB_USER} bash -c "roundup-admin -i ${APP_DIR} install ${TEMPLATE} ${BACKEND} ${OPTIONS}"

\#  tracker conf
CONF=${APP_DIR}/config.ini

\#  disable registration confirmation via email (requires mail server)
OPTION=email_registration_confirmation
sed -i "s|^${OPTION} = yes|${OPTION} = no|" ${CONF}

\#  update web location
OPTION=web
sed -i "s|^${OPTION} = /|${OPTION} = https://support.${APP_HOSTNAME}/support/|" ${CONF}

\# 
\#  HACK start:
\#  need to install to a db that doesn't default to ROW_FORMAT=COMPACT
\#  install into compatible docker db and restore to local mariadb
\# 

HACK=docker
LOCAL_PORT=3306
DOCKER_PORT=3310
DOCKER_HOST=127.0.0.1

\#  set docker db port
OPTION=port
sed -i "1,/^${OPTION} =/s/^${OPTION} =/${OPTION} = ${DOCKER_PORT}/" ${CONF}
\#  docker needs actual 127.0.0.1 ip vs localhost
OPTION=host
sed -i "1,/^${OPTION} = localhost/s/^${OPTION} = localhost/${OPTION} = ${DOCKER_HOST}/" ${CONF}
\#  docker needs "remote" db access so use root
OPTION=user
sed -i "1,/^${OPTION} = ${DB_USER}/s/^${OPTION} = ${DB_USER}/${OPTION} = root/" ${CONF}

\#  docker: get docker
curl -sSL https://get.${HACK}.com/ | sh

\#  docker: start docker db with remove-on-stop flag --rm
CONTAINER=mariadb-10133-container
${HACK} run --rm --name ${CONTAINER} -p ${DOCKER_HOST}:${DOCKER_PORT}:${LOCAL_PORT} -e MYSQL_ROOT_PASSWORD=${DB_PASS} -d mariadb:10.1.33

\#  docker: have to wait for active container
COUNTER=0
until [ "`/usr/bin/${HACK} inspect -f {{.State.Running}} ${CONTAINER}`"=="true" ] || [ ${COUNTER} -gt 30 ]; do
    sleep 1;
    let COUNTER=COUNTER+1
done;
\#  docker: need extra delay
sleep 20;

\#  set roundup password
\#  ${FORMAVID}/bin/initialize/python/roundup.py

\#  initialise tracker using docker db
sudo -H -u ${DB_USER} bash -c "roundup-admin -i ${APP_DIR} initialise ${APP_PASS}"

\#  dump docker db to home to avoid perm issues
${HACK} exec ${CONTAINER} ${MYSQL_DUMP} ${ROOT_CREDS} ${DB_NAME} > ~/${DB_NAME}.sql

\#  remove-on-stop docker
${HACK} stop ${CONTAINER}

\#  TODO: more un-docker here???
systemctl stop ${HACK}
systemctl disable ${HACK}
rm /etc/apt/sources.list.d/${HACK}.list

\#  force ROW_FORMAT=DYNAMIC
sed -i '/^) ENGINE/s/) ENGINE/) ROW_FORMAT=DYNAMIC ENGINE/g' ~/${DB_NAME}.sql

\#  restore dump to local db
mysql ${ROOT_CREDS} ${DB_NAME} < ~/${DB_NAME}.sql

\#  remove dump
rm ~/${DB_NAME}.sql

\#  reset local db port - empty uses default
OPTION=port
sed -i "1,/^${OPTION} = ${DOCKER_PORT}/s/^${OPTION} = ${DOCKER_PORT}/${OPTION} =/" ${CONF}
\#  reset localhost
OPTION=host
sed -i "1,/^${OPTION} = ${DOCKER_HOST}/s/^${OPTION} = ${DOCKER_HOST}/${OPTION} = localhost/" ${CONF}
\#  reset user
OPTION=user
sed -i "1,/^${OPTION} = root/s/^${OPTION} = root/${OPTION} = ${DB_USER}/" ${CONF}

\# 
\#  HACK completed: continue usual
\# 

\#  apply overlay
OVERLAY_DIR=$1/overlay
if [ -d "${OVERLAY_DIR}" ]; then cp -rf ${OVERLAY_DIR}/* /; fi

\#  create etc dir
ETC=/etc/${PACKAGE}
mkdir -p ${ETC}

\#  move tracker configuration to etc
mv ${CONF} ${ETC}/tracker-config.ini
ln -s ${ETC}/tracker-config.ini ${CONF}

\#  don't display credit in popups
for file in *.help*.html _generic.calendar.html; do
    sed -i "s|</head>|<style type='text/css'> #formavid-credit { display: none; }</style>\n  </head>|" ${APP_DIR}/html/${file}
done

\#  create link to local documentation
sed -i "s;Roundup docs</a>;Roundup docs</a>\n    (<a href=\"/docs\" i18n:translate=\"\">local</a>);" ${APP_DIR}/html/page.html

\#  give apache2 access
chown -R www-data:www-data ${APP_DIR}

\#  apache2: create conf
CONF=${ETC}/apache.conf
cat > ${CONF} <<EOF
<VirtualHost *:443>
    ServerName support.${APP_HOSTNAME}
    ServerAdmin webmaster@localhost
    UseCanonicalName Off
    SSLEngine on
    ErrorLog \${APACHE_LOG_DIR}/error.log
    CustomLog \${APACHE_LOG_DIR}/access.log combined
    PythonOptimize On
    Alias /support/docs ${APP_DIR}/docs/html
    Alias /support/theme /var/www/drupal8/themes/${SITENAME}
    AliasMatch ^/support/@@file/(.*) ${APP_DIR}/html/\$1
    AliasMatch ^/support/(?!@@file/)(.*) ${APP_DIR}/dummy.py/\$1
    DocumentRoot ${APP_DIR}
    <Directory ${APP_DIR}>
        Require all granted
        AllowOverride None
        Options None
        AddHandler python-program .py
        PythonHandler roundup.cgi.apache
        PythonOption TrackerHome ${APP_DIR}
        PythonOption TrackerLanguage en
    </Directory>
</VirtualHost>

<VirtualHost *:80>
    ServerName support.${APP_HOSTNAME}
    ServerAdmin webmaster@localhost
    UseCanonicalName Off
    ErrorLog \${APACHE_LOG_DIR}/error.log
    CustomLog \${APACHE_LOG_DIR}/access.log combined
    RedirectPermanent / "https://support.${APP_HOSTNAME}"
</VirtualHost>
EOF

\#  apache2: set conf perms
chmod 0644 ${CONF}

\#  apache2: link conf file
ln -s ${CONF} /etc/apache2/sites-available/${PACKAGE}.conf

\#  apache2: enable conf
a2ensite ${PACKAGE}

\#  log completed
MESSSAGE="Package ${PACKAGE} install has completed."
echo ${MESSSAGE} >> ${LOG}

exit 0

Sorry about all the edits, don't use markdown very often and realized that my cut/paste of code was parsing some slashes and hashtags which is not good if you want things to actually work ;(

BTW, all that is needed for migration is the two steps.

1) Set required globals in new db: QUERY="SET GLOBAL innodb_file_format=Barracuda; SET GLOBAL innodb_file_per_table=ON; SET GLOBAL innodb_large_prefix=1;" mysql ${ROOT_CREDS} ${BATCH_EX} "${QUERY}"

2) sed the dump file from old db: sed -i '/^) ENGINE/s/) ENGINE/) ROW_FORMAT=DYNAMIC ENGINE/g' ~/${DB_NAME}.sql

Should be good to go for direct import after that; don't need the docker "mumbo jumbo" :) It would be wise to keep a copy of the dump until a successful import is verified; as stated, the sed regex may need to be adjusted for an "abby normal" dump.

FWIW

Roundup v15.0 is now released as part of v15.0 - stage 1.

This issue was resolved by installing via pip (as noted in my previous comment).

The full changes to the Roundup since v14.2 can be viewed here. Please also note that the TKLBAM profile has also been updated to reflect the changed install location: https://github.com/turnkeylinux/tklbam-profiles/blob/master/roundup.

@bormanst - FWIW Debian have now backported the option to set innodb_default_row_format = Dynamic for MariaDB databases. So you now simply need to set the config as I have in Foodsoft, here: https://github.com/turnkeylinux-apps/foodsoft/commit/c53c9024026d13d45da0b0f760ff2f6f59dce11a

Thank you very much, that is excellent to know.

On a different note, the http://simpleinvoices.org site appears to be dropping off the map - blank content and dead links, etc. :( The codebase on github still works fine and is being maintained by someone so you probably just have to update a couple url references to the site.

My concerns have led me to start a build for Invoice Ninja https://www.invoiceninja.com/ which is open-source and appears to be a fairly robust solution but I have not test driven it for an honest opinion yet. Some say it can run slow but I'm guessing this is config/hardware related as it runs fine on one cpu in a VM using an old core-2 duo. It does have the defect of product branding that can be white-labeled over for a small fee like many of the so-called free propriety offerings; I don't have an issue with that but you guys may. I'll send a github link to the build file(s) when completed just in case you're interested.

FWIW

On Mon, Nov 26, 2018 at 5:52 PM Jeremy Davis notifications@github.com wrote:

@bormanst https://github.com/bormanst - FWIW Debian have now backported the option to set innodb_default_row_format = Dynamic for MariaDB databases. So you now simply need to set the config as I have in Foodsoft, here: turnkeylinux-apps/foodsoft@c53c902 https://github.com/turnkeylinux-apps/foodsoft/commit/c53c9024026d13d45da0b0f760ff2f6f59dce11a

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/turnkeylinux/tracker/issues/1047#issuecomment-441844922, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH4G1eb80DUqp6T4GjHUAFd7ivqPrIjks5uzH7bgaJpZM4TXJyc .

Thanks for your input @bormanst

FWIW the SimpleInvoices appliance clones from master (on GH): https://github.com/turnkeylinux-apps/simpleinvoices/blob/master/conf.d/clone-repo#L13

It was a while ago now, but I spoke with one of the SimpleInvoices developers/maintainers and he stated that whilst there is no longer any development going on and that they didn't have access to the website/domain/etc, that they intend to maintain it via GH for the foreseeable future. So it's probably safe to continue using it.

However, I agree that InvoiceNinja looks like a great potential appliance. FWIW there is already a "new appliance" issue open for it: https://github.com/turnkeylinux/tracker/issues/715 Although I don't think that development has got very far...

Again, good to know and thanks. I'll post my ninja build info to #715 when completed.

On Wed, Nov 28, 2018 at 3:48 PM Jeremy Davis notifications@github.com wrote:

Thanks for your input @bormanst https://github.com/bormanst

FWIW the SimpleInvoices appliance clones from master (on GH): https://github.com/turnkeylinux-apps/simpleinvoices/blob/master/conf.d/clone-repo#L13

It was a while ago now, but I spoke with one of the SimpleInvoices developers/maintainers and he stated that whilst there is no longer any development going on and that they didn't have access to the website/domain/etc, that they intend to maintain it via GH for the foreseeable future. So it's probably safe to continue using it.

However, I agree that InvoiceNinja looks like a great potential appliance. FWIW there is already a "new appliance" issue open for it: #715 https://github.com/turnkeylinux/tracker/issues/715 Although I don't think that development has got very far...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/turnkeylinux/tracker/issues/1047#issuecomment-442617581, or mute the thread https://github.com/notifications/unsubscribe-auth/AOH4G8bFO2djmDy4-jxzdkog94jllhbRks5uzwTCgaJpZM4TXJyc .

Updated my Apache overlay with your suggestion, modified the build script and it works just fine. Saves a bunch of time not doing the docker hack and allows my app install scripts to run in the background (parallel) which saves even more time :) The docker hack grabbing the db port caused some issues with other apps installing at the same time so...

Thanks again for the heads up, very useful.

p.s. My ninja build info has been posted to #715.

turnkeylinux / tracker

Roundup build: package will not be available in Debian apt for Stretch #1047