search

danb35 / freenas-iocage-nextcloud

Script to create an iocage jail on FreeNAS for the latest Nextcloud 28 release, including Caddy, MariaDB or PostgreSQL, and Let's Encrypt
GNU General Public License v3.0
258 stars 70 forks source link

Missing signing key and basejail options #211

Closed RafalLukawiecki closed 9 months ago

RafalLukawiecki commented 10 months ago
  1. Since the nextcloud.asc key is missing from nextcloud.com (error 404), this PR adds a workaround for obtaining the key directly from a PGP server. The server defaults to pgpkeys.eu because it provides keys with UIDs, which are required by gpg. The server and the key ID can be configured via options.
  2. Added an option for creating a base jail, instead of defaulting to the clone jail, so that updates are easier.
mattmichaels commented 10 months ago

thanks for the fast PR

i tried to run this and got an error creating the jail -

root@freenas:/mnt/NAS-pool/temp/freenas-iocage-nextcloud # sh nextcloud-jail.sh                                                               
JAIL_INTERFACES not set, defaulting to: vnet0:bridge0
Existing Nextcloud config detected... Checking Database compatibility for reinst                                                              all
Database is compatible, continuing...
 is not a valid property!
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_create.py", line 496, in _create_jail
    value, config = iocjson.json_check_prop(key, value, config)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 2501, in json_check_prop
    iocage_lib.ioc_common.logit(
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_common.py", line 107, in logit
    callback(content, exception)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_common.py", line 94, in callback
    raise SystemExit(1)
SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/zfs.py", line 20, in run
    cp.check_returncode()
  File "/usr/local/lib/python3.9/subprocess.py", line 460, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['zfs', 'destroy', '-r', '-Rf', 'SSD/iocage/jails/nextcloud/root']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/iocage_cli/create.py", line 222, in cli
    iocage.create(release, props, count, pkglist=pkglist,
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/iocage.py", line 603, in create
    ioc_create.IOCCreate(
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_create.py", line 101, in create_jail
    return self._create_jail(jail_uuid, location)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_create.py", line 505, in _create_jail
    iocage_lib.ioc_destroy.IOCDestroy().destroy_jail(location)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_destroy.py", line 280, in destroy_jail
    self.__destroy_parse_datasets__(
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_destroy.py", line 256, in __destroy_parse_datasets__
    self.__destroy_dataset__(dataset)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_destroy.py", line 169, in __destroy_dataset__
    ds.destroy(recursive=True, force=True)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/dataset.py", line 98, in destroy
    return destroy_zfs_resource(self.resource_name, recursive, force)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/zfs.py", line 187, in destroy_zfs_resource
    return run([*cmd, resource]).returncode == 0
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/zfs.py", line 22, in run
    raise ZFSException(cp.returncode, cp.stderr)
iocage_lib.zfs.ZFSException: cannot destroy 'SSD/iocage/jails/nextcloud/root': dataset is busy

Failed to create jail

`

mattmichaels commented 9 months ago

getting closer!

go: downloading github.com/quic-go/qtls-go1-20 v0.4.1
go: downloading golang.org/x/text v0.13.0
go: downloading golang.org/x/mod v0.11.0
go: downloading github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572
go: downloading github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1
go: github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b: verifying go.mod: github.com/golang/glog@v0.0.0-20160126235308-23de                    f4e6c14b/go.mod: Get "https://sum.golang.org/lookup/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b": read tcp 192.168.0                    .4:25245->172.253.115.141:443: read: connection reset by peer
2023/12/10 12:30:00 [FATAL] exit status 1
Command: xcaddy build --output /usr/local/bin/caddy --with github.com/caddy-dns/cloudflare failed!
Failed to build Caddy with cloudflare plugin, terminating.
root@freenas:/mnt/NAS-pool/temp/freenas-iocage-nextcloud #
RafalLukawiecki commented 9 months ago

I have only tested it with the AWS Route53 plug in and that worked for me...I am afraid I am not a Caddy expert... Having said that, may I again suggest uncommenting the set -x line and rerunning to see if there is anything obvious? You may need to delete the jail and associated datasets in case they are left around, otherwise the earlier step of building the jail may throw an error.

mattmichaels commented 9 months ago

output:

go: downloading golang.org/x/oauth2 v0.12.0
go: downloading github.com/google/s2a-go v0.1.7
go: downloading cloud.google.com/go/compute/metadata v0.2.3
go: downloading go.opencensus.io v0.24.0
go: downloading github.com/googleapis/enterprise-certificate-proxy v0.2.5
go: downloading cloud.google.com/go/compute v1.23.0
go: downloading google.golang.org/appengine v1.6.7
go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: downloading github.com/jmespath/go-jmespath v0.4.0
2023/12/10 19:46:08 [INFO] exec (timeout=0s): /usr/local/bin/go build -o /usr/local/bin/caddy -ldflags -w -s -trimpath
2023/12/10 19:46:55 [INFO] Build complete: /usr/local/bin/caddy
2023/12/10 19:46:55 [INFO] Cleaning up temporary folder: /tmp/buildenv_2023-12-10-1944.3172767427
+ FILE=latest-27.tar.bz2
+ iocage exec nextcloud fetch -o /tmp https://download.nextcloud.com/server/releases/latest-27.tar.bz2 https://download.nextcloud.com/server/releases/latest-27.tar.bz2.asc
/tmp/latest-27.tar.bz2                                 174 MB    9 MBps    18s
/tmp/latest-27.tar.bz2.asc                             833  B 6612 kBps    00s
+ iocage exec nextcloud fetch -o /tmp https://nextcloud.com/nextcloud.asc
fetch: https://nextcloud.com/nextcloud.asc: Not Found
Command: fetch -o /tmp https://nextcloud.com/nextcloud.asc failed!
+ iocage exec nextcloud gpg --keyserver pgpkeys.eu --recv-key 28806A878AE423A28372792ED75899B9A724937A
gpg: Warning: using insecure memory!
gpg: directory '/root/.gnupg' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key D75899B9A724937A: public key "Nextcloud Security <security@nextcloud.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
+ iocage exec nextcloud gpg --verify /tmp/latest-27.tar.bz2.asc
gpg: Warning: using insecure memory!
gpg: assuming signed data in '/tmp/latest-27.tar.bz2'
gpg: Signature made Thu Nov 23 12:57:02 2023 EST
gpg:                using RSA key 28806A878AE423A28372792ED75899B9A724937A
gpg: Good signature from "Nextcloud Security <security@nextcloud.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 2880 6A87 8AE4 23A2 8372  792E D758 99B9 A724 937A
+ iocage exec nextcloud tar xjf /tmp/latest-27.tar.bz2 -C /usr/local/www/
+ iocage exec nextcloud chown -R www:www /usr/local/www/nextcloud/
+ [ mariadb '=' mariadb ]
+ iocage exec nextcloud sysrc 'mysql_enable=YES'
mysql_enable:  -> YES
+ iocage exec nextcloud sysrc 'redis_enable=YES'
redis_enable:  -> YES
+ iocage exec nextcloud sysrc 'php_fpm_enable=YES'
php_fpm_enable:  -> YES
+ [ 0 -eq 1 ]
+ iocage exec nextcloud cp -f /mnt/includes/php.ini /usr/local/etc/php.ini
+ iocage exec nextcloud chown -R www:www /usr/local/etc/php.ini
+ iocage exec nextcloud cp -f /mnt/includes/redis.conf /usr/local/etc/redis.conf
+ iocage exec nextcloud cp -f /mnt/includes/www.conf /usr/local/etc/php-fpm.d/
+ [ 0 -eq 1 ]
+ [ 1 -eq 1 ]
+ iocage exec nextcloud cp -f /mnt/includes/remove-staging.sh /root/
+ [ 0 -eq 1 ]
+ [ 0 -eq 1 ]
+ [ 1 -eq 1 ]
+ echo $'Copying Caddyfile for Let\'s Encrypt DNS cert'
Copying Caddyfile for Let's Encrypt DNS cert
+ iocage exec nextcloud cp -f /mnt/includes/Caddyfile-dns /usr/local/www/Caddyfile
+ iocage exec nextcloud cp -f /mnt/includes/caddy /usr/local/etc/rc.d/
+ [ mariadb '=' mariadb ]
+ iocage exec nextcloud cp -f /mnt/includes/my-system.cnf /usr/local/etc/mysql/conf.d/nextcloud.cnf
+ iocage exec nextcloud sed -i '' s/yourhostnamehere/www.mjm.host/ /usr/local/www/Caddyfile
+ iocage exec nextcloud sed -i '' s/dns_plugin/cloudflare/ /usr/local/www/Caddyfile
+ iocage exec nextcloud sed -i '' s/api_token// /usr/local/www/Caddyfile
+ iocage exec nextcloud sed -i '' s/jail_ip/192.168.0.4/ /usr/local/www/Caddyfile
+ iocage exec nextcloud sed -i '' s/youremailhere/mattmichaels3@gmail.com/ /usr/local/www/Caddyfile
+ iocage exec nextcloud sed -i '' 's|mytimezone|America/New_York|' /usr/local/etc/php.ini
+ iocage exec nextcloud sysrc 'caddy_enable=YES'
caddy_enable:  -> YES
+ iocage exec nextcloud sysrc 'caddy_config=/usr/local/www/Caddyfile'
caddy_config:  -> /usr/local/www/Caddyfile
+ iocage restart nextcloud
* Stopping nextcloud
  + Executing prestop OK
  + Stopping services OK
  + Tearing down VNET OK
  + Removing devfs_ruleset: 1005 OK
  + Removing jail process OK
  + Executing poststop OK
No default gateway found for ipv6.
* Starting nextcloud
  + Started OK
  + Using devfs_ruleset: 1005 (iocage generated default)
  + Configuring VNET OK
  + Using IP options: vnet
  + Starting services OK
  + Executing poststart OK
+ iocage exec nextcloud touch /var/log/nextcloud.log
+ iocage exec nextcloud chown www /var/log/nextcloud.log
+ iocage exec nextcloud pw usermod www -G redis
+ iocage exec nextcloud chmod 777 /var/run/redis/redis.sock
+ [ true '==' true ]
+ echo 'Reinstall detected, skipping generation of new config and database'
Reinstall detected, skipping generation of new config and database
+ [ mariadb '=' mariadb ]
+ iocage exec nextcloud cp -f /mnt/includes/my.cnf /root/.my.cnf
+ iocage exec nextcloud sed -i '' 's|mypassword|xxxxxxxxx==|' /root/.my.cnf
+ iocage exec nextcloud su -m www -c 'php -f /usr/local/www/nextcloud/cron.php'
Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] No such file or directory in /usr/local/www/nextcloud/lib/private/DB/Connection.php:140
Stack trace:
#0 /usr/local/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1531): OC\DB\Connection->connect()
#1 /usr/local/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1029): Doctrine\DBAL\Connection->getWrappedConnection()
#2 /usr/local/www/nextcloud/lib/private/DB/Connection.php(262): Doctrine\DBAL\Connection->executeQuery('SELECT * FROM `...', Array, Array, NULL)
#3 /usr/local/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(345): OC\DB\Connection->executeQuery('SELECT * FROM `...', Array, Array)
#4 /usr/local/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(280): Doctrine\DBAL\Query\QueryBuilder->execute()
#5 /usr/local/www/nextcloud/lib/private/AppConfig.php(418): OC\DB\QueryBuilder\QueryBuilder->execute()
#6 /usr/local/www/nextcloud/lib/private/AppConfig.php(184): OC\AppConfig->loadConfigValues()
#7 /usr/local/www/nextcloud/lib/private/AppConfig.php(374): OC\AppConfig->getApps()
#8 /usr/local/www/nextcloud/lib/private/legacy/OC_App.php(803): OC\AppConfig->getValues(false, 'installed_versi...')
#9 /usr/local/www/nextcloud/lib/private/Server.php(736): OC_App::getAppVersions()
#10 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\Server->OC\{closure}(Object(OC\Server))
#11 /usr/local/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#12 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\Container->offsetGet('OC\\Memcache\\Fac...')
#13 /usr/local/www/nextcloud/lib/private/ServerContainer.php(171): OC\AppFramework\Utility\SimpleContainer->query('OC\\Memcache\\Fac...', true)
#14 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\ServerContainer->query('OC\\Memcache\\Fac...')
#15 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(193): OC\AppFramework\Utility\SimpleContainer->get('OC\\Memcache\\Fac...')
#16 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server))
#17 /usr/local/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#18 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\Container->offsetGet('OCP\\ICacheFacto...')
#19 /usr/local/www/nextcloud/lib/private/ServerContainer.php(171): OC\AppFramework\Utility\SimpleContainer->query('OCP\\ICacheFacto...', true)
#20 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\ServerContainer->query('OCP\\ICacheFacto...')
#21 /usr/local/www/nextcloud/lib/private/Server.php(1130): OC\AppFramework\Utility\SimpleContainer->get('OCP\\ICacheFacto...')
#22 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\Server->OC\{closure}(Object(OC\Server))
#23 /usr/local/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#24 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\Container->offsetGet('OCP\\Lock\\ILocki...')
#25 /usr/local/www/nextcloud/lib/private/ServerContainer.php(171): OC\AppFramework\Utility\SimpleContainer->query('OCP\\Lock\\ILocki...', true)
#26 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\ServerContainer->query('OCP\\Lock\\ILocki...')
#27 /usr/local/www/nextcloud/lib/private/Server.php(2088): OC\AppFramework\Utility\SimpleContainer->get('OCP\\Lock\\ILocki...')
#28 /usr/local/www/nextcloud/lib/private/Files/View.php(107): OC\Server->getLockingProvider()
#29 /usr/local/www/nextcloud/lib/private/Server.php(470): OC\Files\View->__construct()
#30 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\Server->OC\{closure}(Object(OC\Server))
#31 /usr/local/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#32 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\Container->offsetGet('OC\\Files\\Node\\H...')
#33 /usr/local/www/nextcloud/lib/private/ServerContainer.php(171): OC\AppFramework\Utility\SimpleContainer->query('OC\\Files\\Node\\H...', true)
#34 /usr/local/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\ServerContainer->query('OC\\Files\\Node\\H...')
#35 /usr/local/www/nextcloud/lib/private/Server.php(1490): OC\AppFramework\Utility\SimpleContainer->get('OC\\Files\\Node\\H...')
#36 /usr/local/www/nextcloud/lib/base.php(635): OC\Server->boot()
#37 /usr/local/www/nextcloud/lib/base.php(1196): OC::init()
#38 /usr/local/www/nextcloud/cron.php(43): require_once('/usr/local/www/...')
#39 {main}
Command: su -m www -c php -f /usr/local/www/nextcloud/cron.php failed!
+ iocage exec nextcloud crontab -u www /mnt/includes/www-crontab
+ iocage fstab -r nextcloud /mnt/NAS-pool/temp/freenas-iocage-nextcloud/includes /mnt/includes nullfs rw 0 0
Successfully removed mount from nextcloud's fstab
+ echo 'Installation complete!'
Installation complete!
+ [ 0 -eq 1 ]
+ echo 'Using your web browser, go to https://www. to log in'
Using your web browser, go to https://www. to log in
+ [ true '==' true ]
+ echo 'You did a reinstall, please use your old database and account credentials'
You did a reinstall, please use your old database and account credentials
+ echo ''

+ [ 0 -eq 1 ]
+ [ 1 -eq 1 ]
+ echo $'You have obtained your Let\'s Encrypt certificate using the staging server.'
You have obtained your Let's Encrypt certificate using the staging server.
+ echo 'This certificate will not be trusted by your browser and will cause SSL errors'
This certificate will not be trusted by your browser and will cause SSL errors
+ echo $'when you connect.  Once you\'ve verified that everything else is working'
when you connect.  Once you've verified that everything else is working
+ echo 'correctly, you should issue a trusted certificate.  To do this, run:'
correctly, you should issue a trusted certificate.  To do this, run:
+ echo '  iocage exec nextcloud /root/remove-staging.sh'
  iocage exec nextcloud /root/remove-staging.sh
+ echo ''
RafalLukawiecki commented 9 months ago

First of all, it looks like the PR has worked for you. You are well past the stage of having successfully downloaded and validated the .tar with the Nextcloud key, which was successfully obtained from the key server.

It looks like you have a different issue now, perhaps worth opening a new bug report here? Something along the lines of "Script fails on reinstall". It looks like the offending line is iocage exec nextcloud su -m www -c 'php -f /usr/local/www/nextcloud/cron.php' and I wonder if this is an issue of a password mismatch. In general, since you are doing a reinstall over an existing Nextcloud, perhaps you may want to test the script doing a fresh install? Either way, this is a new issue.

danb35 commented 9 months ago

Thanks for the help in taking care of this. In the future, it's best to limit PRs to a single issue, but in this case both changes are beneficial so I just merged it.