Radicale Issue

[vkarpyuk]

Hi Guys. I have two issues with radicale calendar and I guess somehow they are related.

Firstly, when I go /calendars/#/ on the web interface it allows me to create new calendars but when I try to create a new event it doesn't allow me.

Logs from /var/logs/supervisor: [2020-11-10 06:50:02 +0000] [1148/Thread-73] [ERROR] An exception occurred during OPTIONS request on '/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-10 06:50:43 +0000] [1148/Thread-74] [ERROR] An exception occurred during OPTIONS request on '/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-10 06:57:32 +0000] [1148/Thread-75] [ERROR] An exception occurred during OPTIONS request on '/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-10 06:58:59 +0000] [1148/Thread-76] [ERROR] An exception occurred during OPTIONS request on '/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-10 07:00:01 +0000] [1148/Thread-77] [ERROR] An exception occurred during OPTIONS request on '/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-10 07:00:26 +0000] [1148/Thread-78] [ERROR] An exception occurred during OPTIONS request on '/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-10 07:00:46 +0000] [1148/Thread-79] [ERROR] An exception occurred during OPTIONS request on '/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 17:06:03 +0000] [1148/Thread-80] [ERROR] An exception occurred during PUT request on '/viktor@kubrikdigital.com/KUBRIK/ede8a1af-df99-441a-99c9-42b2fab04015.ics': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 17:08:10 +0000] [1148/Thread-81] [ERROR] An exception occurred during REPORT request on '/viktor@kubrikdigital.com/KUBRIK/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 17:08:15 +0000] [1148/Thread-82] [ERROR] An exception occurred during PUT request on '/viktor@kubrikdigital.com/KUBRIK/65f8c24c-0522-4525-aaae-3115a25efeaf.ics': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 17:08:17 +0000] [1148/Thread-83] [ERROR] An exception occurred during PUT request on '/viktor@kubrikdigital.com/KUBRIK/4a2650ce-6070-4549-bc1f-f3f28c2c6ccd.ics': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 17:08:18 +0000] [1148/Thread-84] [ERROR] An exception occurred during PUT request on '/viktor@kubrikdigital.com/KUBRIK/baf2b953-e7d8-43e1-b670-9c4f6c1aaf35.ics': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 17:08:18 +0000] [1148/Thread-85] [ERROR] An exception occurred during PUT request on '/viktor@kubrikdigital.com/KUBRIK/30374ed1-6124-434a-8c25-730e69c39bab.ics': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 18:37:31 +0000] [1148/Thread-86] [ERROR] An exception occurred during PUT request on '/viktor@kubrikdigital.com/KUBRIK/70ec3b05-8fff-415e-846f-2f25cc09cec4.ics': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 18:37:52 +0000] [1148/Thread-87] [ERROR] An exception occurred during MKCALENDAR request on '/viktor@kubrikdigital.com/PERSONAL': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 18:37:52 +0000] [1148/Thread-88] [ERROR] An exception occurred during REPORT request on '/viktor@kubrikdigital.com/PERSONAL/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 18:39:32 +0000] [1148/Thread-89] [ERROR] An exception occurred during REPORT request on '/viktor@kubrikdigital.com/KUBRIK/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 18:39:32 +0000] [1148/Thread-90] [ERROR] An exception occurred during REPORT request on '/viktor@kubrikdigital.com/PERSONAL/': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 18:39:37 +0000] [1148/Thread-91] [ERROR] An exception occurred during PUT request on '/viktor@kubrikdigital.com/PERSONAL/74e5e14b-de4e-4743-af67-b5c908afb006.ics': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 18:39:38 +0000] [1148/Thread-92] [ERROR] An exception occurred during PUT request on '/viktor@kubrikdigital.com/PERSONAL/32c0aea7-d411-4982-9cc8-d656fdcbbc46.ics': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 18:39:39 +0000] [1148/Thread-93] [ERROR] An exception occurred during PUT request on '/viktor@kubrikdigital.com/PERSONAL/add49c39-3a65-42a3-b5b1-5191d7fe2a09.ics': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?' [2020-11-11 18:39:39 +0000] [1148/Thread-94] [ERROR] An exception occurred during PUT request on '/viktor@kubrikdigital.com/PERSONAL/da23c6f9-75cb-4d1e-91e6-346c8e0f0f96.ics': Error in section 'owners-access' of rights file '/etc/radicale/rights': Bad value substitution: option 'collection' in section 'owners-access' contains an interpolation key 'login' which is not a valid option name. Raw value: '%(login)s(/.)?'

Second issue: When I try to connect to CalDav it doesn't allow from any caldav client:

Third issue and I assume it's related to the previous one: When I try to import the ICS it fails with 500.

Logs: /var/log/uwsgi/app/modoboa_instance.log: mail.kubrikdigital.com [pid: 3152172|app: 1|req: 838/1899] 50.99.234.98 () {64 vars in 1467 bytes} [Thu Nov 12 02:33:44 2020] POST /api/v1/user-calendars/1/events/import_from_file/ => generated 870 bytes in 169 msecs (HTTP/1.1 500) 5 headers in 181 bytes (1 switches on core 0)

Thanks in Advance!

[tonioo]

@vkarpyuk Can you check which Radicale version you're using please? And also, can you give information about your Modoboa setup?

[vkarpyuk]

Hi @tonioo,

Thanks for your prompt reply!

Here you are the installation information

Regards, Viktor

[tonioo]

@vkarpyuk And what about Radicale? (the server part)

[vkarpyuk]

@tonioo what do you mean by the server part? It was installed as part of the installation of the modoboa server.

[tonioo]

@vkarpyuk Radicale is a caldav/carddav server used to store calendars and contacts on your server. It is automatically installed by the installation script and the package should be installed in /srv/radicale/env. Looking at your logs, I think you're using an old version, which is possible if you installed Modoboa some time ago.

[vkarpyuk]

@tonioo here you are the installation path, it looks correct.

I installed the latest modoboa version that was available at the time of installation. I previously had a problem and had to reinstall the server. In fact, I switched to a new server so had to reinstall everything. It's weird because it's no longer a couple of months since I installed it.

[vkarpyuk]

Hi @tonioo,

Should I upgrade radicale to https://radicale.org/3.0.html? In case it's yes, will it be integrated with the modoboa server?

Thanks.

[vkarpyuk]

Guys, do you have any clue of what should I do in order to make the radicale work as expected? Should I reinstall it? If yes, how do I run the installation script without affecting the modiboa installation itself?

I would appreciate your response!

Thanks in advance!

[MeGaMoV]

Hi @vkarpyuk,

After a lot of weird debugs during 1 entire week, i have finally found how to fix this.

1 : Update / Downgrade radicale server 👨‍💻

source /srv/radicale/env/bin/activate pip install radicale==2.1.12

You will ask me "Why 2.1.12 ?", because it's the latest version working for modoba. If you take a look at : https://github.com/Kozea/Radicale/blob/3.0.6/NEWS.md#upgrade-checklist They changed the /etc/radicale/rights file format, and it just not work with the autogeneration of modoboa.

2 : Again some errors 🥴

It's not finished, because you will have another error if you take a look at https://github.com/modoboa/modoboa-installer/issues/325 and https://github.com/Kozea/Radicale/issues/1061

You need to add / edit some lines in /etc/radicale/config file

ADD This (to avoid errors of missing config values for dovecot) auth_host = localhost auth_port = 10000

EDIT The storage filesystem : [storage] type = multifilesystem #radicale_storage_by_index_fields = dtstart, dtend, uid, summary

3 : Restart and test 😎

supervisorctl restart radicale

All of this worked on my side, I hope it will be same for you

[vkarpyuk]

Hi @MeGaMoV!

Thank you very much for your detailed explanation! Fingers crossed, I hope this will work for me too! Will keep you posted once I'm done with the re-installation!

Regards, Viktor

[vkarpyuk]

Hi @MeGaMoV,

tried to change the radicale version to 2.1.12 but still the same erros.

Regards, Vik

[MeGaMoV]

Hi @vkarpyuk,

Please confirm us what version of radicale server do you have with

source /srv/radicale/env/bin/activate
pip list

[vkarpyuk]

Hi @MeGaMoV! Thanks for help with commands. Here the result of pip list:

[MeGaMoV]

Then, it's weird because you have radicale 3.0 parsing errors..

@vkarpyuk please do this command : supervisorctl restart radicale

And then live check supervisor logs with : tail -f /var/log/supervisor/radicale*.log

And try to create a new event / calendar in the panel to show us what you saw with the tail command.

Thanks

[vkarpyuk]

This is how my /radicale config file looks like(just in case)

# This file was automatically installed on 2020-10-06T04:27:16.575517
# -*- mode: conf -*-
# vim:ft=cfg

# Config file for Radicale - A simple calendar server
#
# Place it into /etc/radicale/config (global)
# or ~/.config/radicale/config (user)
#
# The current values are the default ones

[server]

# CalDAV server hostnames separated by a comma
# IPv4 syntax: address:port
# IPv6 syntax: [address]:port
# For example: 0.0.0.0:9999, [::]:9999
#hosts = 127.0.0.1:5232

# Daemon flag
#daemon = False

# File storing the PID in daemon mode
#pid =

# Max parallel connections
#max_connections = 20

# Max size of request body (bytes)
#max_content_length = 10000000

# Socket timeout (seconds)
#timeout = 10

# SSL flag, enable HTTPS protocol
#ssl = False

# SSL certificate path
#certificate = /etc/ssl/radicale.cert.pem

# SSL private key
#key = /etc/ssl/radicale.key.pem

# CA certificate for validating clients. This can be used to secure
# TCP traffic between Radicale and a reverse proxy
#certificate_authority =

# SSL Protocol used. See python's ssl module for available values
#protocol = PROTOCOL_TLSv1_2

# Available ciphers. See python's ssl module for available ciphers
#ciphers =

# Reverse DNS to resolve client address in logs
#dns_lookup = True

# Message displayed in the client when a password is needed
#realm = Radicale - Password Required

[encoding]
                                                                                                                                                                                                             1,3           
# Encoding for responding requests
#request = utf-8

# Encoding for storing local collections
#stock = utf-8

[auth]

auth_host = localhost
auth_port = 10000

# Authentication method
# Value: none | htpasswd | remote_user | http_x_remote_user
type = radicale_dovecot_auth

# Htpasswd filename
# htpasswd_filename = users

# Htpasswd encryption method
# Value: plain | sha1 | ssha | crypt | bcrypt | md5
# Only bcrypt can be considered secure.
# bcrypt and md5 require the passlib library to be installed.
# htpasswd_encryption = plain 

# Incorrect authentication delay (seconds)
#delay = 1

auth_socket = /var/run/dovecot/auth-radicale

[rights]

# Rights backend
# Value: none | authenticated | owner_only | owner_write | from_file
type = from_file

# File for rights management from_file
file = /etc/radicale/rights

[storage]

# Storage backend
# Value: multifilesystem
type = multifilesystem
#type = radicale_storage_by_index
#radicale_storage_by_index_fields = dtstart, dtend, uid, summary

# Folder for storing local collections, created if not present
filesystem_folder = /srv/radicale/collections

# Lock the storage. Never start multiple instances of Radicale or edit the
# storage externally while Radicale is running if disabled.
#filesystem_locking = True

# Sync all changes to disk during requests. (This can impair performance.)
# Disabling it increases the risk of data loss, when the system crashes or
# power fails!
#filesystem_fsync = True

# Delete sync token that are older (seconds)
#max_sync_token_age = 2592000

# Close the lock file when no more clients are waiting.
# This option is not very useful in general, but on Windows files that are
# opened cannot be deleted.
#filesystem_close_lock_file = False

# Command that is run after changes to storage
# Example: ([ -d .git ] || git init) && git add -A && (git diff --cached --quiet || git commit -m "Changes by "%(user)s)
#hook =

[web]

# Web interface backend
# Value: none | internal
type = none

[logging]

# Logging configuration file
# If no config is given, simple information is printed on the standard output
# For more information about the syntax of the configuration file, see:
# http://docs.python.org/library/logging.config.html
#config = /etc/radicale/logging

# Store all environment variables (including those set in the shell)
#full_environment = False

# Don't include passwords in logs
#mask_passwords = True

[headers]

# Additional HTTP headers
#Access-Control-Allow-Origin = *

I created a new calendar event, however, couldn't find any log related to the event creation in this log (tail -f /var/log/supervisor/radicale*.log)

[MeGaMoV]

Can you check for us if your radicale server is running ? supervisorctl status radicale

Looks like it couldn't start because of the error I mentioned earlier

[vkarpyuk]

I've checked the status before, and it works.

[MeGaMoV]

Ok so everything looks fine now. You will not see any info logs into this file because it only shows errors ;)

[vkarpyuk]

Hi @MeGaMoV,

Thanks for all your support! After all done, I still have the same issue. My calendar is not being synced to radicale :(. As of now, I'm using thunderbird but only its local storage.

I Will keep you posted once I have more details. It looks like there's something else missing on the server-side of modoboa/radicale configuration.

Regards, Vik

[MeGaMoV]

Hi @vkarpyuk ,

Is the internal modoboa/radicale web calendar working without any issue ?

[vkarpyuk]

Hey @MeGaMoV,

No, it doesn't.

[MeGaMoV]

Ok then, what do you have when checking the logs ? tail -f /var/log/supervisor/radicale*.log

[vkarpyuk]

It doesn't log any error there.

[vkarpyuk]

[7fa7de38a740] ERROR: An exception occurred during server startup: Failed to load storage module 'radicale_storage_by_index': No module named 'radicale.storage.multifilesystem'; 'radicale.storage' is not a package [7fe4900f4740] ERROR: An exception occurred during server startup: Failed to load storage module 'radicale_storage_by_index': No module named 'radicale.storage.multifilesystem'; 'radicale.storage' is not a package [7fdb56161740] ERROR: An exception occurred during server startup: Failed to load storage module 'radicale_storage_by_index': No module named 'radicale.storage.multifilesystem'; 'radicale.storage' is not a package [7f4a2d8c3740] ERROR: An exception occurred during server startup: Failed to load storage module 'radicale_storage_by_index': No module named 'radicale.storage.multifilesystem'; 'radicale.storage' is not a package [7f236feff740] ERROR: An exception occurred during server startup: Failed to load storage module 'radicale_storage_by_index': No module named 'radicale.storage.multifilesystem'; 'radicale.storage' is not a package [7ff5fb6ca740] ERROR: An exception occurred during server startup: Failed to load storage module 'radicale_storage_by_index': No module named 'radicale.storage.multifilesystem'; 'radicale.storage' is not a package

[MeGaMoV]

Ok it's the errors you've got last day. What errors you have into your web browser ? Is still 500 error ?

[vkarpyuk]

Yes,

[MeGaMoV]

Do you get errors with the personnal calendar or only the shared one ?

[vkarpyuk]

Either of them.

[MeGaMoV]

Can you see your account / mail in the radicale's right file ? /etc/radicale/rights ?

And in the first line check the time that it was generated like this (normally <= 2 mins ago) :

[vkarpyuk]

Here you are the snapshot of /etc/radicale/rights file:

# Rights management file for Radicale
# This file was generated by Modoboa on 2020-11-22 21:10:04.252721
# DO NOT EDIT MANUALLY!

[sa-admin-acr]
user = admin
collection = .*
permission = rw

[sa-viktor@kubrikdigital.com-acr]
user = viktor@kubrikdigital.com
collection = .*
permission = rw

[da-admin@kubrikdigital.com-to-kubrikdigital.com-acr]
user = admin@kubrikdigital.com
collection = kubrikdigital.com/user/.*
permission = rw

[MeGaMoV]

Take a look here https://modoboa-radicale.readthedocs.io/en/latest/setup.html

For me, i need to force ( I don't know why :D ) :

Manually :

source /srv/modoboa/env/bin/activate
cd /srv/modoboa/instance
python manage.py generate_rights --force

Verify your cron line :

nano /etc/cron.d/modoboa

[vkarpyuk]

Did as you mentioned. Regenerated the rights file and restarted redicale service(just in case). Tried again to create a new event on the modoboa calendar app and again, it's failing with 500.

[MeGaMoV]

Is the date and elements on /etc/radicale/rights file changed ?

[vkarpyuk]

Oh sorry! I've forgotten to mention that the file's date had changed.

[vkarpyuk]

Just realized that it also fails with 500 for the initial page load of the events. Look at the console log

[vkarpyuk]

@MeGaMoV,

That's very very weird! I deleted previous calendars: one personal and the other one shared. Created a new Test Calendar and I was able to create events without failures.

[MeGaMoV]

Yes maybe creating a new one with this configuration wil work now, is it ok now ? Same for Contacts ?

[vkarpyuk]

The contacts were working perfectly. Still have a problem with shared calendars.

[vkarpyuk]

@MeGaMoV, it still gives me 500 when I try to post a shared calendar event.

[tonioo]

@vkarpyuk Shared calendars are not working as expected because Radicale does not really support them. I need to figure out how to solve this...

[tonioo]

Looks I opened a bug regarding the access file format a few months ago (https://github.com/modoboa/modoboa-radicale/issues/53) but I totally forgot to release the fix!

[vkarpyuk]

Hi @tonioo,

Thanks for the heads up, that would be a great feature to have! Also, I would recommend you to review the personal calendar functionality.

I have one use case where my user can create/fetch events in the modoboa web calendar. However, another user can create the event in the modoboa web calendar, it gets persisted and accessible from thunderbird, but it fails with 500 in the modoboa calendar. It looks like there's a bug also.

Regards, Viktor

[tonioo]

@vkarpyuk Looks there was a permission issue with default configuration. I've pushed a fix and I'll release a new version soon.

[tonioo]

Could you try version 1.5.1 please? And don't forget to regenerate the rights file.

[vkarpyuk]

Hi @tonioo,

Thanks a lot! I will try at the weekend. I can't compromise the server today. Are there any specific steps to be taken? I've never upgraded before. Any comments are welcome!

Thanks in advance!

[tonioo]

Take a look at the documentation: https://modoboa.readthedocs.io/en/latest/upgrade.html#extensions.

[PatTheMav]

To pick this up, I am running modoboa-radicale v1.5.1 and the current version of Radicale and the rights file seems to work fine (I tested with an outdated virtualenv before which created a mismatch between the updated format and the older(!) version of Radicale).

But now I get 403 Forbidden errors when creating contacts, but not when creating calendar events. Created calendar events don't appear when revisiting the Calendar page either however.

Another issue is that contacts are created in the database even when the sync with Radicale fails, which then creates contacts that can neither be edited nor deleted (as Radicale will always throw a 404 error as the vcf files haven't been created.

EDIT: I managed to fix those errors by deleting the collection for the specific identity after Radicale worked fine with a different identity, but now I get a "409 Conflict" error when saving contacts.

[tonioo]

@PatTheMav Do you see more information in radicale logs?

[PatTheMav]

Not that I can see - I ran Radicale with debug mode to see the live output - the login via dovecote-auth works, it checks against the rights file successfully, the vCard also looks right, but then it just says that there is a conflict and returns 409.

I still have to run some tests with other identities to rule out that something might be broken with this particular identity.

[PatTheMav]

@tonioo Better late then never, I've managed to find the root cause of my issues:

• Root cause for the 403 Forbidden error is a missing .Radicale.props file that correctly identifies a folder inside the collection as an address book or a calendar. Adding such a file with {"tag": "VADDRESSBOOK"} as its sole content fixed the issue for adding/editing contacts.
• Modoboa might need to abort the contact creation process when it receives an error from radicale (if radicale is used) - in my testing I ended up with lots of orphaned contacts that you cannot delete, because the deletion itself fails (as radicale can't find the corresponding object) - instead I have to clean up these orphaned contacts in the database manually
• The 409 Conflict disappeared once I reactivated the store-by-index module, but I can imagine it had the same root cause as the 403 Forbidden error.

I brush this up to me futzing around with the radicale collections when I re-built my mail server and I just imported the databases but as I didn't create a new mailbox in modoboa itself, I had to recreate the collection folder structure myself and didn't set up the props files correctly.