Safihre
commented
3 years ago @hgy59 Is there an overview of changes I need to make to get DSM7 compatibility for my package (sabnzbd)?
Open hgy59 opened 3 years ago
Safihre
commented
3 years ago @hgy59 Is there an overview of changes I need to make to get DSM7 compatibility for my package (sabnzbd)?
publicarray
commented
3 years ago @Safihre I suggest to start with the Wiki DSM7.0-Beta
For your package you may only need to change ${SYNOPKG_PKGDEST}/var to ${SYNOPKG_PKGVAR} and knowing set_syno_permissions won't be doing anything in DSM 7. Just inspect the log files if there are additional permission errors. Unfortunately I can't test DSM7 stuff for now, as I'm waiting for my replacement unit (possibly something wrong with the power/mobo inside my unit failed.) :cry:
Safihre
commented
3 years ago So how do we handle the permissions?
publicarray
commented
3 years ago Generally you don't. Since you don't have root. DSM will chown the folders for you (using the package name) if you need other folders they need to be given in the resource worker or the user has to add the package from the DSM settings (look for "internal users" category) to a folder.
Safihre
commented
3 years ago Can we maybe update set_syno_permissions to do the service worker stuff?
publicarray
commented
3 years ago Under the DSM 7 restrictions I don't have a clear idea how that would work. Not saying it's impossible but even if done it wouldn't be backwards compatible anyway. If you're willing to contribute sure. But once you read DSM Developer Guide 7 0 Beta in full I think you might reconsider. IMHO the work required (if it's even possible with the new restrictions) to make it work is not worth the time.
I encourage you look at the transmission package. It allows only the user to choose a Shared Folder through the wizard.
The reality is that the NAS now manages the permissions, and packages only get a few selected knobs to play with. The permissions are in the user hands through the file manager and control center as "internal user" (IHMO that's a good thing. So malicious software can't just upload random files to a remote server)
Safihre
commented
3 years ago Sabnzbd also has a wizard already, but I don't quite see from the Transmission example what needs to be changed about it to make it work. Is it something in the name of the variable? I have not really been part of the whole DSM 7 story, so I don't know what's already impmented in the framework. My own NAS can't even run it (old model), so I will admit that I am not very motivated to really dive deep just to update my 1 package. 😕
DigitalBox98
commented
3 years ago On DSM7.0, rabbitmq is failing to start due to incorrect service_postinst(). As indicate in #4539, the installer logs must be redesigned and fixed for DSM7.0 to allow many packages to work again :)
daxcore
commented
3 years ago update/repairing after update to dsm7 of umurmur fails because of
3) Does not install due to required root privilege
publicarray
commented
3 years ago @daxcore This is expected. Did you read our Readme.md? If you where to read the table above then you would see that everything is very much work in progress on the master branch. (no packages are published yet, but you can compile your own).
hgy59
commented
3 years ago @publicarray Thanks for the wiki-page DSM-7.0-Beta.
As we get closer to DSM 7.0, I wonder when we will rename this page (remove the beta). As all the existing links to this page will be broken by a rename, we might better start with a new wiki page for DSM 7.
We could wait for such a page until DSM 7 (final or RC1) is released and document validated facts only.
What would you prefer?
publicarray
commented
3 years ago I propose that relevant stuff in DSM-7.0-Beta is moved/copied to their respective wiki page (Permission Management, Generic Service Support etc.) and that we create a quick DSM7 upgrade guide or a quick dsm7 reference for developers (IMHO the detailed wikis tend to quite long, I think some stuff is repetitive and or It's hard to find a specific thing. It's good to have them tough especially for newcomers but a quick dot-point list like the DSM-7.0-beta is more digestible for a quick lookup, kinda like a cheat sheet. What do you think?
hgy59
commented
3 years ago To give an overview of packages that need additional data migration, I have added the column Data Folder.
This is for all folders that need extra data migration (config files). The target/var folder must not be listed here as it is already migrated in the current installer code.
publicarray
commented
3 years ago Maybe we should migrate the target/etc the same way we migrate target/var ?
hgy59
commented
3 years ago Maybe we should start to set os_max_ver (as in #4567) for all DSM<=6 packages to something like os_max_ver=6.999 .
Hopefully this will limit the packages shown in DSM7 package center to those that are published for DSM7.
th0ma7
commented
3 years ago @hgy59 perhaps something closer to os_max_ver=6.9-39999.
Sadly this field is not well documented in the developer guide (even in previous I found).
Will it actually block it from showing in the package center? I guess that could be tested?
That could be quite easy to add to my current PR such as seting from the spksrc.tc-vers.mk file such as:
if ($(word 1,$(subst ., ,$(TC_VERS)),5)
TC_OS_MAX_VER=5.9-6999
else if ($(word 1,$(subst ., ,$(TC_VERS)),6)
TC_OS_MAX_VER=6.9-39999
endifThen in spksrc.spk.mk change my snipet for:
ifneq ($(strip $(OS_MAX_VER)),)
@echo os_max_ver=\"$(OS_MAX_VER)\" >> $@
else ifeq ($(strip $(TC_OS_MAX_VER)),)
@echo os_max_ver=\"$(TC_OS_MAX_VER)\" >> $@
# If require kernel modules then limit
# 'os_max_ver' to point releases only
else ifeq ($(strip $(REQUIRE_KERNEL)),1)
@echo os_max_ver=\"$(word 1,$(subst ., ,$(TC_VERS))).$(word 2,$(subst ., ,$(TC_VERS)))-$$(expr $(TC_BUILD) + 10)\" >> $@
endifLastly add to spksrc.tc.mk:
@echo TC_BUILD := $(TC_BUILD)
@echo TC_OS_MIN_VER := $(TC_OS_MIN_VER)
@echo TC_OS_MAX_VER := $(TC_OS_MAX_VER)
@echo TC_ARCH := $(TC_ARCH)That could be quite easy to add to my current PR such as seting from the
spksrc.tc-vers.mkfile such as:
I prefer to test this with my local repo as we have to wait up to 48h for the repo on synocommunity.com until new packages appear in the package center in DSM.
th0ma7
commented
3 years ago @hgy59 let me know if you want me to adjust #4567 or prefer another PR if deemed working out as expected.
hgy59
commented
3 years ago Unfortunatly this does not work with the current implementation of spkrepo and there is already an open issue https://github.com/SynoCommunity/spkrepo/issues/63. Your solution in #4567 will not work either.
th0ma7
commented
3 years ago Your solution in #4567 will not work either.
@hgy59 The intent of that particular PR was not related to spkrepo but rather making sure that after an upgrade the package would be disabled as now incompatible with the new kernel.
hgy59
commented
3 years ago @th0ma7 I do not understand what you mean by disabled. As the spkrepo does not care about the third part of the version number and always delivers the highest build number of Major.Minor version this will look like this:
say we have only such versions of package in the repo:
Now you have devices with different DSM Versions installed
With the current spkrepo implementation all these devices will get the same version of package downloaded in the DSM package center
You cannot force to download package-6.2-24922 for DSM 6.2.2 when package-6.2-25556 is not compatible with DSM 6.2.2 due to different kernel version.
th0ma7
commented
3 years ago @hgy59 from a spkrepo point of view, you are most probably right. I don't know if there is anything that can be done to add such field so it is understood somehow from the package center which would be quite useful.
I am rather referring to the actual package, already installed/running on your DSM: I want to make sure that it becomes disabled (or removed) pre-post applying a DSM upgrade that affects the kernel.
With this I assume that, lets say you are running DSM 6.2.3-25423 and using a compatible kernel package, it will stop working when you install DSM 6.2.4-25556 upgrade due to the os_max_ver variable in already installed package.
Dark2004
commented
3 years ago Dear @hgy59 how can i install these packages (noarch) on my NAS with DSM 7.0? i do not find any source?
thanks
hgy59
commented
3 years ago Dear @hgy59 how can i install these packages (noarch) on my NAS with DSM 7.0? i do not find any source?
As long as those packages are not published, you need to install the spk files manually in the DSM Package Center of your Diskstation.
You can download packages created with github build action of the related PR or on master branch for merged PRs.
Or you can create those packages in the development environment in the spk/{package} folder with
make TCVERSION=7.0
sam43434
commented
3 years ago offical release of DSM 7 for those who hasn't seen the news final build comes out June 29th,
th0ma7
commented
3 years ago ffmpeg + tvheadend + chromaprint + comskip now published.
BenjV
commented
3 years ago I did some testing on DSM 7 and here are some findings and suggestions. For packages that need to share files, the old (pre DSM 7) framework supported the group sc-downloads, but because as of DSM 7 the installation scripts can only run with package privs so this solution does not function any more.
My suggestion would be to use the next privilege file for those packages who needs to share files with other packages.
{
"defaults": {
"run-as": "package"
},
"username": "sc-<packagename>",
"groupname": "synocommunity"
}All those package are then using the group e.g."downloads" (don't use sc-downloads, because that can already exists and will create problems).
Also I would suggest to use a resource file like this:
{
"data-share": {
"shares": [
{
"name": "{{wizard_share_name}}",
"permission": {
"rw": [
"sc-<package name>"
]
}
}
]
},
"port-config": {
"protocol-file": "app/<package name>.sc"
}
}This will create a new share owned by the package user and the group "downloads"
The input for the sharename can be gotten from a install_uifile (or hard coded if you which). Here an example of a install_uifile to do so.
[
{
"step_title": "Basic configuration",
"items": [
{
"type": "textfield",
"desc": "Used Share for downloads",
"subitems": [
{
"key": "wizard_share_name",
"desc": "sharename",
"defaultValue": "<package name>"
}
]
}
]
}
]Combined with a postinstall script like this:
if [ ! -d "$SYNOPKG_PKGDEST_VOL/$SYNOPKG_PKGNAME/downloads" ]
then
mkdir "$SYNOPKG_PKGDEST_VOL/$SYNOPKG_PKGNAME/downloads"
chmod 770 "$SYNOPKG_PKGDEST_VOL/$SYNOPKG_PKGNAME/downloads"
fiThis configuration will create a new share with group privileges for the group "downloads" and a subfolder name "downloads" with the correct privs.
sam43434
commented
3 years ago unsure if this is the final release thats suppose to come out today but its higher then the RC version https://archive.synology.com/download/Os/DSM/7.0-41890
Marcoevich
commented
3 years ago @sam43434 yeah this one is the final release.
BenjV
commented
3 years ago @hgy59 , @ymartin59 , @th0ma7 and @publicarray What is your opinion on my suggestion to use the same groupname in the privilege file for packages that need to be able to access files from other packages instead of creating the group sc0download in the scripts? And also create a data-share with the resource file to place the files they need to share.
And if you agree with this proposal, what is the best name for this group?
Safihre
commented
3 years ago Oh no... I thought this whole problem of shared folders between applications was already implemented 😔 seems like yesterday when we had to fix this exact same problem for DSM 6. @BenjV so your proposal is allowed by DSM, so that multiple resources can be shared between applications?
BenjV
commented
3 years ago @Safihre Yes I tested it on DSM 7, and this way the "hidden" package users will all be using the same "hidden" group. And within the resource file you can create a share which is owned by this "hidden" user:group combo and if you add group access to that share in the postinstall phase with a chmod command, all packages from that group will have access.
So the most important issue at the moment is to agree on this methode and to decide the name of that group. We could use the name "download", but not the old name "sc-download" because that will give conflicts.
hgy59
commented
3 years ago In the data share definition, we can define multiple users, so we can additionally give access to the community groups (@sc-download, @sc-media, ...) for such shares.
"data-share": {
"shares": [{
"name": "<share-name>",
"permission": {
"ro": ["<user-name>", ...],
"rw": ["<user-name>", ...]
},
"once": "<once>"
}, ...]
}... postinstall phase with a chmod command, all packages from that group will have access.
Wouldn't chmod fail in DSM7 as now restricted (or I got this wrong)?
So the most important issue at the moment is to agree on this methode and to decide the name of that group. We could use the name "download", but not the old name "sc-download" because that will give conflicts.
We have sc-download and sc-media (any others?). I would much rather keep them...
BenjV
commented
3 years ago @th0ma7 That share is created and owned by the "hidden" package owner and so the script has privs to chmod them. Default it is created with only rw privs for the owner so you have to add rw privs for its group as well
If you use an existing name "sc-download" and people upgrade from DSM 6 to DSM 7 you get problems. The existing share will be renamed to something like "sc-download_PKG" and that can create problems if the old name is used by the end-user.
@hgy59 You can only add a share during installation of a package, not later on. So adding access later on won't work. Adding an existing name with another package will cause a rename of the share created by a previous package. And if you use the "once" keyword, it won't do anything at all if the share already exists, so no extra privs are added.
Using the same group for all those packages is clearly the easiest way to solve this. After all, groups exists for sharing privs.
hgy59
commented
3 years ago You can only add a share during installation of a package, not later on. So adding access later on won't work.
This behavior is when setting "once" : "true" in the data share definition.
If you set this to "false" or omit this, the share permissions are adjusted/updated at every service start.
th0ma7
commented
3 years ago Would it make sense to create a SynoCommunity core package to manage this specifically (and potentially other items alike)? Then make all packages depends on it by default with DSM7.
BenjV
commented
3 years ago @hgy59 I just tested that and you are correct, so that could work too. In that case just the privs are added.
If this methode would be used, a share with a fixed name should be created for those packages and not leave it to the user via a install_ui wizard, because we must be sure the same share name is always used. If you use the group method that would not be necessary, because every share created via this method would be part of the same group.
Safihre
commented
3 years ago @hgy59 @BenjV Isn't the whole wizard stuff removed in DSM 7? At least that's what I understood from the discussions previously?
BenjV
commented
3 years ago @Safihre I tested it and you are correct. Synology removed it from the final DSM 7 release.
My testing was first done on the release candidate and their it was still functioning, in de final DSM 7 release you cannot install a package with a wizard_ui file on board anymore.
In the release candidate it still was available and functioning, but in the final version it is gone. Also in de beta version of the development guide it was still present, but in the new development guide that chapter is gone.
@hgy59, @ymartin59, @th0ma7 and @publicarray So a fixed share must be created by all those packages, via the resource file. Anyone wants to propose a name?
th0ma7
commented
3 years ago Here's a proposal:
volume1/
└── SynoCommunity
├── download
└── mediaSo far we discuss new installations, but what about existing shares and packages? How do users grant access to the packages to use other/existing folders? Is it obvious how to do it? How will it work after packages are upgraded, can they still access their old shares? (my ancient Synology won't get DSM 7 update)
BenjV
commented
3 years ago @th0ma7 Fine with me, but you cannot choose the volume. Via the resource file the share is always created on the same volume as the volume where the package is installed (mostly volume1). This can create problems if end-users are installing different packages on different volumes, but that is not to be avoided and will rarely occur I think.
I would still suggest to use a common group name in the privilege file for all packages, that would create maximal flexibility. Lets name that group also "SynoCommunity"
cgilis
commented
3 years ago So far we discuss new installations, but what about existing shares and packages? How do users grant access to the packages to use other/existing folders? Is it obvious how to do it? How will it work after packages are upgraded, can they still access their old shares? (my ancient Synology won't get DSM 7 update)
Imo it's clear how to grant access
BenjV
commented
3 years ago @Safihre End-users can login as admin and give privs to the "system internal users" on shares. The problem for packages is that they cannot assign privs during installation, because the scripts do not run as root anymore.
And current installed packages should still work, although they all must be updated because all actions from the scrips that need root privs will not work and will stop installation and/or upgrading. DSM 7 is very picky and will refuse to install with the message that the package is corrupted. When I upgraded to DSM 7 there was only one of several packages that survived it ( it was git) all other installed package disappeared from the package center.
hgy59
commented
3 years ago @hgy59 @BenjV Isn't the whole wizard stuff removed in DSM 7? At least that's what I understood from the discussions previously?
No, wizards are not removed (and the discussion was withing the maintainers group of this community) I have a WIP with minio and the wizard including the page to choose share (name), access-key and secret key is still shown with DSM 7.0-41890.
hgy59
commented
3 years ago So a fixed share must be created by all those packages, via the resource file. Anyone wants to propose a name?
Even when synology might remove installation wizards some time, we should only use real shared folders defined by the share-name.
We still have to verify that an existing share is used and the users defined in the resource file of the package are added to that share.
But: This will allow the users to create the share on any volume before installation.
real shared folders mean folders, that are created similar to "Control Panel - Shared Folders - Create". Such folders can only be managed by the "Control Panel - Shared Folders" control - it is not possible to delete those on the command line (even with root access).
My preference would be to use the named shared folders sc-download and sc-media (i.e. name it with the name of the synocommunity common group names).
BenjV
commented
3 years ago If you add a share in the resource file it is just like all other shares a real share. The only difference is that the "system internal user" of the package is the owner of that share.
You should not use existing shares, that will cause the existing share to be renamed.
All shares can be remove via the commandline with the synoshare --del
th0ma7
commented
3 years ago My preference would be to use the named shared folders
sc-downloadandsc-media(i.e. name it with the name of the synocommunity common group names).
Agreeing with you on using the sc- prefix. Also, usage of singular, always.
With it I wonder if the SynoCommunity top folder is really needed:
volumeX/
└── SynoCommunity
├── sc-download
└── sc-mediaor
volumeX/
├── sc-download
└── sc-mediaIn the first case you only need one share and two folders. The second case needs two shares to be created.
In my opinion it is better not to create unnecessary shares. Rember that shares have more overhead then folders, especialy on the network.
Overview of the status of packages for DSM7
Successfull build/install/run is tested with x64 only (unless otherwise stated).
As of may 2021 all pure CLI packages are ready to be published for DSM7. Please regard DSM7 packages still as
beta. The synocommunity package repository does not support beta packages perTCVERSIONso it is not possible to mark packages as beta for DSM7 only. Packages that other packages depend on are distingueshed in bold. Those should be published as early as possible.hassiodsm7-packagesbranchRemarks: 1) Avoid link and usage of
/usr/local/{package-name}2) Migrate mysql tomariadb-103) Does not install due to required root privilege 4) Installer uses unsupported function likeset_unix_permissions, syno_remove_user, syno_group_create, syno_group_remove, syno_user_add_to_group, set_syno_permissions, syno_user_add_to_legacy_group5) Requires a dependent package that might not be published yet. 6)SERVICE_WIZARD_SHAREis not supported for DSM7. Shared folders must be declared in resources and must be the name of the share (without volume). 7) Packages that integrate into WebStation (web apps, web services) must register via WebStation webapi or use a Package Worker. 8)target/varis not listed here, as migration is already implemented in current installer. 9) Invalid version number. DSM7 validates$(SPK_VERS)-$(SPK_REV)with regex[^\d+(\.\d+){0,5}(-\d+)?$]. So only digits and dots are allowed inSPK_VERS. 10)SERVICE_EXEneeds migration toSERVICE_COMMAND. 11) As Python2 of DSM 7 comes neigther withpipnorvirtualenvit looks like we have to deploy python2 for DSM 7 (at least for the single package ffsync that will never be ported to Python 3 as it is about to be migrated to Rust).⛔ Packages that require root privilege to run will not be compatible with DSM7. ⌛ Packages that need root previlege for installation will not be compatible until synology allows permission settings for specific installation steps. ⚙️ Packages that need access to USB devices. synology might drop support for USB devices.