500 on createing calendar

[capocasa]

Connecting worked fine with DAVx5, but there were no calendars pre-created, and I got the following error on creating one with the DAVx5 Gui:

Oct 27 13:31:21 martin env[29284]: [2020-10-27 13:31:21 +0100] [29284/Thread-19] [ERROR] An exception occurred during MKCOL request on '/carlo/c35c4a77-598c-47b2-ad32-91ec7f141a52/': Expecting value: line 1 column 1 (char 0)

Not sure what to do from here.

[capocasa]

Getting a similar error when attempting to "find calendars" using Evolution. Just found debug mode, here's the trace:

Traceback (most recent call last): File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/init.py", line 105, in call status, headers, answers = self._handle_request(environ) File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/init.py", line 277, in _handle_request environ, base_prefix, path, user) File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/propfind.py", line 374, in do_PROPFIND self._encoding) File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/propfind.py", line 65, in xml_propfind for item, permission in allowed_items: File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/propfind.py", line 313, in _collect_allowed_items for item in items: File "/usr/local/lib/python3.7/dist-packages/radicale_remind.py", line 187, in discover for filename in adapter.get_filesnames(): File "/usr/local/lib/python3.7/dist-packages/remind.py", line 218, in get_filesnames self._update() File "/usr/local/lib/python3.7/dist-packages/remind.py", line 214, in _update self._reminders = self._parse_remind(self._filename) File "/usr/local/lib/python3.7/dist-packages/remind.py", line 78, in _parse_remind for month in loads(stdout.decode('utf-8')): File "/usr/lib/python3.7/json/init.py", line 348, in loads return _default_decoder.decode(s) File "/usr/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode

[jspricke]

I don't think creating calenders is supported (haven't tested). Do you have a remind file in at the location remind_file is pointing to?

[capocasa]

I see!

Well OK my first feedback is then that this should go in the docs- only one calendar as per the defined remind file, no creating calendars.

My second feedback would be- better error messages for common operations needed, so the error message would tell you this on trying to create a calendar. Lots of detail work, I know! But that's "discoverabily" of how it's supposed to work, so that would be the best way to improve usability I think. Other common caldav operations seem to be "browse for calendars". Evolution seems like a distant way to test.

A pointer to radicale debug mode would also help for testing things.

OK so I made double sure that there is an existing remind file in place, and now the error changed. (Edit: This is on reading calendar)

Traceback (most recent call last):
  File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/__init__.py", line 105, in __call__
    status, headers, answers = self._handle_request(environ)
  File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/__init__.py", line 277, in _handle_request
    environ, base_prefix, path, user)
  File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/report.py", line 288, in do_REPORT
    lock_stack.close)
  File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/report.py", line 153, in xml_report
    multistatus))
  File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/report.py", line 139, in retrieve_items
    for name, item in collection.get_multi(get_names()):
  File "/usr/local/lib/python3.7/dist-packages/radicale_remind.py", line 71, in get_multi
    return ((x[0], self._convert(x)) for x in self.adapter.to_vobjects(self.filename, hrefs))
AttributeError: 'NoneType' object has no attribute 'to_vobjects'

The remind file is

REM 27 Oct 2020 MSG Foo

I hope this helps with the docs!!! Thanks for helping me out, as well!!!

[jspricke]

Sorry, had to work. What did you set filesystem_folder and remind_file to in your radicale config? (you can put any names but the number of directories is important, you can send it to my mail address as well if you want)

[capocasa]

Of course, no problem!!! I appreciate the support.

I tried both of these:

filesystem_folder = /home/carlo/Sync/Calendar
remind_file = /home/carlo/Sync/Calendar/remind

filesystem_folder = /home
remind_file = /home/carlo/Sync/Calendar/remind

[jspricke]

Ok, the second variant is really close to what I have in my config. As a start, how much Python knowledge do you have? Could you get me some debug data of how discover() is called here: https://github.com/jspricke/radicale-remind/blob/master/radicale_remind.py#L166 ? I can guide you through, just want to ask first to not write to much ;).

Regarding your other feedback, feel free to open new bugs so we don't loose them (though this is a plugin to Radicale, so I'm a bit limited in printing error messages, for example).

[capocasa]

I do python web dev with flask and uwsgi, so I can follow dev instructions, thanks for asking! Sure, I can provide debug info.

What would really help would be a little background information on how remind-radicale is intended to be used- you could answer the questions in #19 and place the answers directly into the docs, I think that might save time for you to do it this way.

Also, do you use some kind of caldav debugging client, such as a command line client, that I could use so I'm sure we're both running the same commands?

[jspricke]

Can you add logger.warning(f'remind debug: {path}') in line 181 and write the output here? See #19 and #20 for the other questions.

[capocasa]

I made the edit, started radicale in debug mode, and performed a "refresh calendar list" on DAVx5.

Here is the output! a.log

Here are my remind files, one main one called remind and two includes, in a zip archive a.zip

[capocasa]

I think I made some progress. Using the new curl command, I realized that the username part was actually the username from the home directory. I have my remind files in a separate directory called /home/carlo/Sync/Calendar/remind. I symlinked remind to ~/.reminders, and called /carlo/.reminders, but I keep getting 403s on the server.

[2020-10-30 18:10:11 +0100] [15681/Thread-6] [INFO] GET response status for '/carlo/.reminders' in 0.013 seconds: 403 Forbidden

[jspricke]

You could disable the authentication (and bind Radicale to localhost, just in case) to work around that one. I will try to reproduce your folder layout later today.

[capocasa]

Cool! I'll try that. You could also provide a canonical set of remind files that work for you, and also include those as part of the docs, so you always have a working setup to start with after installing radicale-remind.

[jspricke]

The problem is somewhere in the Radicale config, not the Remind files. I will continue debugging next week, the weather is too good ;).

Am 31. Oktober 2020 14:58:50 MEZ schrieb Carlo Capocasa notifications@github.com:

Cool! I'll try that. You could also provide a canonical set of remind files that work for you, and also include those as part of the docs, so you always have a working setup to start with after installing radicale-remind.

[capocasa]

Of course! I'm just real happy with your help!!! And that you are productively using my ignorance to benefit the documentation is really very cool, I love being useful because I'm clueless. ;)

[jspricke]

I tried to reproduce your issue but failed.

• Which version of Radicale do you use? (I currently use upstream master)
• Which versions of radicale-remind and python-remind do you use? (I use git, but the last releases should be fine as well).

To test a minimal setup:

• Use the Radicale config from this projects readme.
• Set ssl to false.
• Set auth type to none.
• hosts = 127.0.0.1:5232

Test with curl with http (without s). This works for me with your Remind files (with a.zip/remind as ~/.reminders). Note that the main file has no reminders.

[capocasa]

Minimal setup I still get the same error, but I'm sure the server is hitting the file now. I set it up in /tmp/carlo/.reminders with no auth, as specified. I get the error:

[2020-11-04 12:39:11 +0100] [28196/Thread-1] [ERROR] An exception occurred during GET request on '/carlo/.reminders/': Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/__init__.py", line 105, in __call__
    status, headers, answers = self._handle_request(environ)
  File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/__init__.py", line 277, in _handle_request
    environ, base_prefix, path, user)
  File "/home/carlo/.local/lib/python3.7/site-packages/radicale/app/get.py", line 77, in do_GET
    item = next(self._storage.discover(path), None)
  File "/usr/local/lib/python3.7/dist-packages/radicale_remind.py", line 197, in discover
    if filename in adapter.get_filesnames():
  File "/usr/local/lib/python3.7/dist-packages/remind.py", line 218, in get_filesnames
    self._update()
  File "/usr/local/lib/python3.7/dist-packages/remind.py", line 214, in _update
    self._reminders = self._parse_remind(self._filename)
  File "/usr/local/lib/python3.7/dist-packages/remind.py", line 78, in _parse_remind
    for month in loads(stdout.decode('utf-8')):
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I am using remind 03.01.16, as came with Debian buster, and python-remind 0.15.0 that came with pip.

I also tried with remind 03.03.01 and python-remind 0.15.0 that came with arch linux and pip for python 3.8, respecively.

Result is similar:

[2020-11-04 12:46:26 +0100] [264960/Thread-8] [DEBUG] Sanitized script name: ''
[2020-11-04 12:46:26 +0100] [264960/Thread-8] [DEBUG] Sanitized path: '/carlo/.reminders/'
[2020-11-04 12:46:26 +0100] [264960/Thread-8] [INFO] Successful login: 'carlo'
[2020-11-04 12:46:26 +0100] [264960/Thread-8] [ERROR] An exception occurred during GET request on '/carlo/.reminders/': Expecting value: line 4 column 1 (char 327)
Traceback (most recent call last):
  File "/home/carlo/.local/lib/python3.8/site-packages/radicale/app/__init__.py", line 105, in __call__
    status, headers, answers = self._handle_request(environ)
  File "/home/carlo/.local/lib/python3.8/site-packages/radicale/app/__init__.py", line 276, in _handle_request
    status, headers, answer = function(
  File "/home/carlo/.local/lib/python3.8/site-packages/radicale/app/get.py", line 77, in do_GET
    item = next(self._storage.discover(path), None)
  File "/home/carlo/.local/lib/python3.8/site-packages/radicale_remind.py", line 195, in discover
    if filename in adapter.get_filesnames():
  File "/usr/lib/python3.8/site-packages/remind.py", line 218, in get_filesnames
    self._update()
  File "/usr/lib/python3.8/site-packages/remind.py", line 214, in _update
    self._reminders = self._parse_remind(self._filename)
  File "/usr/lib/python3.8/site-packages/remind.py", line 78, in _parse_remind
    for month in loads(stdout.decode('utf-8')):
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 4 column 1 (char 327)

[jspricke]

I'm sorry that you have to go through all this and I really hope we can fix this and improve the project :).

The Debian buster error is expected, as python-remind needs at least 03.03.01, as documented here: https://github.com/jspricke/python-remind#installation You could get the new version from Debian testing.

The arch one is interesting, python-remind can't parse the json provided by remind. Can you paste the output of: remind -ppp15 -b2 -y -df ~/.reminders 2020-11-04

Also, would you be available for a videocall on Friday or the weekend? I would really like to have this fixed.

[capocasa]

I see! May I suggest you introduce some kind of assertion or exception in python-remind with an error message explaining that? That would then be a huge time saver for users- discoverable configuration. See https://github.com/jspricke/python-remind/issues/11

Here is the output you mentioned for arch: remind-arch.log

It's okay, thanks for making the software! Glad to help. As you noticed I am really very keen on getting remind files on my phone and I suspect I will be using it for a very long time. I never looked back since I switched to todo.txt.

[jspricke]

Thanks for the bug report, I was just to lazy to implement that, will do.

the log parses fine for me, can you add this at line 78 of remind.py? open('/tmp/remind.log', 'w').write(stdout.decode('utf-8')) and attach the file?

[capocasa]

Hahaha I just installed it on a cloud test instance I have running with debian unstable. I was going to give you reproduce it for you there so you can debug it autonomously- I don't think I can swing doing remote hands until to completion- there it actually worked!!!

[jspricke]

Nice! Yeah, I'm testing with unstable as well..

[capocasa]

Oh good! OK then I think the root cause here is simply lack of testing on different systems.

Do you think you could spin up a buster image locally or on hetzner or something and see if you can install from pip? I would recommend doing the same for two or three different distributions. If you do buster it should also work on Ubuntu, but maybe a fedora based one? Do people still use that? ;) Of course arch is also great but I'm biased on that one. That should lead you to various subtle issues that should also allow you to improve your heuristics for cross version python coding-

[jspricke]

Sure I could spin up all systems but honestly that would rather be a job for a package maintainer and radicale-remind is maybe even too small to attract enough people for that.

Let's do it the other way round, which OS do you want to use later? I can try to reproduce your bugs there and fix them.

[capocasa]

Hey, we're in luck! All I needed to do on debian buster was install remind from source, which was pretty straight forward. There was nothing wrong with python-remind or remind-radicale on python 3.7, which was what I feared.

You're probably right about testing on lots of systems yourself, I was just worried there were going to be more bugs to find than either of us can handle with remote hands. Thanks for your willingness to try it with one system, though!

I've got my remind caldav set up, I'm going to work with it and report back.

[capocasa]

OK so now I've got my .reminders, holidays and main calendars displaying in DAVx5, but I get a 404 for main an holidays in radicale.

So if you have any ideas I can try feel free, or it may be more efficient to proceed to try and reproduce on your own buster image.

[2020-11-05 10:41:08 +0100] [11051/Thread-12] [INFO] PROPFIND response status for '/carlo/holidays/' with depth '0' in 0.152 seconds: 207 Multi-Status
[2020-11-05 10:41:08 +0100] [11051/Thread-13] [INFO] PROPFIND request for '/carlo/main/' with depth '0' received from 192.168.178.22 using 'DAVx5/3.3.5-gplay (2020/10/24; dav4jvm; okhttp/4.9.0) Android/10'
[2020-11-05 10:41:08 +0100] [11051/Thread-13] [DEBUG] Request headers:
{'CONTENT_LENGTH': '265',
 'CONTENT_TYPE': 'application/xml; charset=utf-8',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT_ENCODING': 'br,gzip',
 'HTTP_ACCEPT_LANGUAGE': 'en-DE, en;q=0.7, *;q=0.5',
 'HTTP_CONNECTION': 'Keep-Alive',
 'HTTP_DEPTH': '0',
 'HTTP_HOST': 'martin:5232',
 'HTTP_USER_AGENT': 'DAVx5/3.3.5-gplay (2020/10/24; dav4jvm; okhttp/4.9.0) '
                    'Android/10',
 'PATH_INFO': '/carlo/main/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '192.168.178.22',
 'REMOTE_HOST': '',
 'REQUEST_METHOD': 'PROPFIND',
 'SCRIPT_NAME': '',
 'SERVER_NAME': 'martin.capo.casa',
 'SERVER_PORT': '5232',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SOFTWARE': 'WSGIServer/0.2',
 'wsgi.errors': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>,
 'wsgi.file_wrapper': <class 'wsgiref.util.FileWrapper'>,
 'wsgi.input': <_io.BufferedReader name=4>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}
[2020-11-05 10:41:08 +0100] [11051/Thread-13] [DEBUG] Sanitized script name: ''
[2020-11-05 10:41:08 +0100] [11051/Thread-13] [DEBUG] Sanitized path: '/carlo/main/'
[2020-11-05 10:41:08 +0100] [11051/Thread-13] [DEBUG] Request content:
<?xml version="1.0"?>
<propfind xmlns="DAV:" xmlns:CS="http://calendarserver.org/ns/">
  <prop>
    <supported-report-set />
    <CS:getctag />
    <sync-token />
  </prop>
</propfind>

[2020-11-05 10:41:08 +0100] [11051/Thread-13] [WARNING] remind debug: /carlo/main/
[2020-11-05 10:41:08 +0100] [11051/Thread-13] [DEBUG] Response content:
The requested resource could not be found.
[2020-11-05 10:41:08 +0100] [11051/Thread-13] [INFO] PROPFIND response status for '/carlo/main/' with depth '0' in 0.004 seconds: 404 Not Found

[jspricke]

I need some more information:

• Did davx5 automatically discover the calendars and generated the URLs?
• Do you still have the remind file linked to ~/.reminders and the included in a different folder?
• Can you try moving the into the same folder as a test? I'm not sure how well python-remind works with links, but we can work on the if that is the problem.
• Which version of Radicale do you use? I had some problems with the rights management of it in some versions.
• Does it work if you put events in the main remind file? (As a test)

[jspricke]

@capocasa did you have time to look into this already?

[capocasa]

Not quite! I will soon!

[capocasa]

Hello again!

Didn't report back for a while- would theoretically have time now, but after a little thinking- now, I don't mean to be rude by making demands on your time, because I know this is an open source project and I really don't mean to be the guy making unreasonable demands on OSS developer time. So this is just an observation...

I think the project is simply not a state where it can be efficiently improved by me forwarding you my error messages. That is for difficult-to-reproduce edge case bugs.

At this state, if you wish to have more users, you need to get some Linux virtual machines, and test your software on those- and/or code more defensively so things are more robust- preferably both.

Because if I keep forwarding you error messages when the software is at this stage- then you are quite simply doing just that, testing the software on a different linux distribution, you are just having my remote hands in the development loop, and that is a very inefficient way of doing things- for you even more than for me! I predict that simply too many issues will come up this way for the effort to be sustainable.

So please kindly decide if you would like to invest the time required to turn your project into one that has a certain level of stability on linux systems. If you do, I will be more than happy to test on my system and iron out some edge cases. Now if you don't that's perfectly legitimate, I just want to point out what I believe would be required to move the project forward. In any case, thanks for making it! I'd love to look into the codebase myself and do pull requests instead, I just already have a whole lot of pull requests I intended to look into as well so I have no idea when I'd be able to make time for that... but I will let you know if I do.

Just to give you the current state! I hope I was understandable, and questions and further comments are quite welcome.

[jspricke]

Thanks for coming back to this! I totally agree that this project needs works and more testers, but I don't understand why you think that's related to the operating system. I don't see anything related to that in the issue (only Remind being old, but that was documented) and the software is rather system agnostic. What I really need is users with different setups and creative ways to use this software, I don't think I can simulate this myself.

Would you be in for a live debugging session in Jitsi?

[capocasa]

why you think that's related to the operating system

Because a different operating system will affect any program you run on it in all sorts of ways, and they are impossible to foresee. Therefore, your system simply cannot be considered system-agnostic until you tested it on a handful of different ones.

Here is a list of things that could possibly crash your program if they are different by default on a different OS:

• Python version (including patch version)
• Python library version
• Supporting software versions, such as the web server
• File permissions
• Existing users
• System policies (e.g. apparmor, default firewall settings)

In other words, it is insufficient to follow best practices for an operating-system agnostic programming style. You must actually test your software on different operating systems until it can be considered to run on each one, and you must document the installation steps performed on each one, and exactly which configuration was used on each one. Without this, there is so much guesswork involved as to make installing your software very difficult and painful, as it was for me.

Now, we could theoretically do the live debugging session- but as explained, what you're really doing is working the kinks out running your software in a environment where it never ran before, but just using be as an intermediary. That's not efficient, because I'm basically remote hands.

Instead: Do you think I could spin up a debian stable image in the cloud, give you the key, and you get it to run there? If you can do this, I am confident that I can transfer the configuration to my live server and it will work. And then I would be happy to write some nice instructions, put them into the readme- then buster users can be reasonably expected to be able to install it.

The reason I can't do that myself is that I don't have "creator eyes"- little issues for you take hours for me.

[jspricke]

why you think that's related to the operating system

Because a different operating system will affect any program you run on it in all sorts of ways, and they are impossible to foresee. Therefore, your system simply cannot be considered system-agnostic until you tested it on a handful of different ones.

This is true in general but note that radicale-remind is only a plugin for Radicale, so we can assume a working Radicale.

Here is a list of things that could possibly crash your program if they are different by default on a different OS:

• Python version (including patch version)

This is compatible starting with Python 3.5.

• Python library version
• Supporting software versions, such as the web server
• File permissions
• Existing users
• System policies (e.g. apparmor, default firewall settings)

These are all general topics not relevant to this plugin, sorry.

In other words, it is insufficient to follow best practices for an operating-system agnostic programming style. You must actually test your software on different operating systems until it can be considered to run on each one, and you must document the installation steps performed on each one, and exactly which configuration was used on each one. Without this, there is so much guesswork involved as to make installing your software very difficult and painful, as it was for me.

I'm sorry that you hade to go through this but I all I can say is "works for me".

Now, we could theoretically do the live debugging session- but as explained, what you're really doing is working the kinks out running your software in a environment where it never ran before, but just using be as an intermediary. That's not efficient, because I'm basically remote hands.

I disagree here, what we are working out is the specifics of your setup which I don't know.

Instead: Do you think I could spin up a debian stable image in the cloud, give you the key, and you get it to run there? If you can do this, I am confident that I can transfer the configuration to my live server and it will work. And then I would be happy to write some nice instructions, put them into the readme- then buster users can be reasonably expected to be able to install it.

Can you set it up such that it produces the error you have above? Than I'm happy to investigate myself.

[capocasa]

Okay I think I understand where you're coming from. It would appear to me that both our positions are based on assumptions that might or might not be true, but I would be happy to set up a reproduction environment for you so we can find out what's going on. I wouldn't rue the time to do a remote debugging session as well, I just unironically really suck at reliably transmitting commands and forwarding replies for any length of time, so it would be super helpful to me if I could plug you in directly. I'll let you know when it's done, and thanks for all your support for your project!

[capocasa]

I have sent you access data to a disposable server via email.

I have followed the pypi setup instructions and performed:

# as root
apt install remind abook
# as user rr
pip install pip install radicale-remind
# created ~/.config/radicale/config
mkdir ~/.abook
radicale

[2021-05-03 13:59:40 +0200] [2007] [CRITICAL] An exception occurred during server startup: ('storage', 'remind_lookahead_month')

remind -v
REMIND 03.03.01