search

etesync / server

The Etebase server (so you can run your own)
https://www.etesync.com
GNU Affero General Public License v3.0
1.54k stars 75 forks source link

v0.5.0 is unable to open database file #59

Closed plague-doctor closed 3 years ago

plague-doctor commented 3 years ago

Etesync-server version from branch v0.3.0 works like a charm, but build from branch v0.5.0 cannot start and throws:

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 207, in get_new_connection
    conn = Database.connect(**conn_params)
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 121, in inner_run
    self.check_migrations()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 459, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 53, in __init__
    self.build_graph()
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 216, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 77, in applied_migrations
    if self.has_table():
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 55, in has_table
    with self.connection.cursor() as cursor:
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 207, in get_new_connection
    conn = Database.connect(**conn_params)
django.db.utils.OperationalError: unable to open database file

What am I missing?

tasn commented 3 years ago

It's a weird error, and I'm not sure why you are getting that exactly, but just so you know: you can't just update to v0.5.0. You need to run the new server in parallel and run the migration from one of the apps (as explained in the EteSync 2.0 migration instructions). I'll clarify the instructions in the README now.

tasn commented 3 years ago

Did what I mention solve your issue about running both in parallel (or just following the migration path).

plague-doctor commented 3 years ago

I have tried once and it was successful, but soon after web client started to tell me it is unable to decrypt... I wasn't able to share my calendars either (which is a critical for me)... It was most likely my fault, but have no time to dig deeper at the moment. I use etesync in docker so will need to fiddle a bit to run two different instances the same time...

tasn commented 3 years ago

You don't really need to run both instances in parallel. I just improved the migration instructions for self-hosted to better explain that: https://github.com/etesync/server/#updating-from-version-050-or-before

Web client has a new address now https://pim.etesync.com Please take a look at the migration guide for more information: https://www.etesync.com/user-guide/migrate-v2/

plague-doctor commented 3 years ago

Ok. I have done few more attempts:

  1. Created a new etesync-server docker instance (release 0.5.1).
  2. Created a new account through https://etesync.my.domain/admin
  3. Made migration using etesync app as per your description. Worked!

4a. Then I go to https://etesync-web.my.domain (release 0.6.0) and try to log into the new account... Fails with Wrong password for user. and this is what I see on console:

etesync-server    | [05/Nov/2020 14:36:17] "GET / HTTP/1.1" 200 297
etesync-server    | [05/Nov/2020 14:36:53] "POST /api/v1/authentication/login_challenge/ HTTP/1.1" 200 125
etesync-server    | Unauthorized: /api/v1/authentication/login/
etesync-server    | [05/Nov/2020 14:36:57] "POST /api/v1/authentication/login/ HTTP/1.1" 401 58

4b. When I try to connect to the old account from the same etesync-web, then it spits out:

<h1>Not Found</h1><p>The requested resource was not found on this server.</p>
  1. From what I can see, the app on the phone works fine. I can add/edit/remove calendar events, etc... Just no luck with the web interface.

More tests:

  1. I have created user through https://etesync.my.domain/admin
  2. No migration, just tried to login through https://etesync-web.my.domain. It errors out with: User not properly init

Looks like I need some help with the web gui...

tasn commented 3 years ago

You have a working etesync 2.0 account that works on Android but not on the web? Are you using the latest etesync web? If so, what version? Could you maybe come to https://www.etesync.com/community-chat/ at some point for some real time debugging?

User not properly init means that the account was never initialised, so it's very odd that it works on Android.

tasn commented 3 years ago

Also, after investigating it elsewhere, it looks like the original issue (can't open db file) is just because it was trying to use an etesync v1 db instead of a new one.

plague-doctor commented 3 years ago

You have a working etesync 2.0 account that works on Android but not on the web?

That's right.

Are you using the latest etesync web? If so, what version?

As stated in 4a: release 0.6.0 I will try later today from master branch.

User not properly init means that the account was never initialised, so it's very odd that it works on Android.

I get User not properly init when I try to connect from etesync-web right after I create the user via https://etesync.my.domain/admin, not after the android migration. I have not tried to connect to such user from android yet.

plague-doctor commented 3 years ago

Tested with etesync-web (master branch)

I have stopped old etesync so I do not cross-polinate by a mistake.

tasn commented 3 years ago

Ah, I figured out what's up, at least for "Test 2" (not sure about test 1 yet). I was thinking about latest from git when I was thinking about latest, not latest release. I now released 0.6.1 that should fix it (was already released in master). What happens if you use this newly created (and setup with etesync-web) account with Android? Also fails? I gotta ask, are you absolutely sure the password is correct? Could you please maybe try on a test user with a basic password like 12345678 to make sure it's the same?

4b. You mean etesync v1? If so, that's expected. The new client only supports EteSync 2.0.

plague-doctor commented 3 years ago

Sorry @tasn, but I cannot see a new release 0.6.1... Are we talking about the same repo? You mean this: https://github.com/etesync/etesync-web Currently for tests I use:

I am sure I use the right password (copy/paste from password manager).

And can you elaborate on:

What happens if you use this newly created (and setup with etesync-web) account with Android?

How can I setup a new account with etesync-web? If you mean create a new using https://etesync-server/admin and then login using https://etesync-web, then I am unable to login - User not properly init

tasn commented 3 years ago

Ah it was tagged but not "released". Got I hate github's UI around this. Anyhow, v0.6.1 is now there with a github release too.

How can I setup a new account with etesync-web?

I meant using using signup (after creating the account from /admin), though you can now, with 0.6.1 also just login and it'll set your password.

plague-doctor commented 3 years ago

Booo... User not properly init etesync-server log:

etesync-server    | 0 static files copied to '/var/www/static', 165 unmodified.
etesync-server    | Watching for file changes with StatReloader
etesync-server    | [07/Nov/2020 20:06:30] "OPTIONS /api/v1/authentication/login_challenge/ HTTP/1.1" 200 0
etesync-server    | [07/Nov/2020 20:06:30] "POST /api/v1/authentication/login_challenge/ HTTP/1.1" 200 125
etesync-server    | [07/Nov/2020 20:06:34] "OPTIONS /api/v1/authentication/login/ HTTP/1.1" 200 0
etesync-server    | Unauthorized: /api/v1/authentication/login/
etesync-server    | [07/Nov/2020 20:06:34] "POST /api/v1/authentication/login/ HTTP/1.1" 401 58
etesync-server    | Unauthorized: /api/v1/authentication/login_challenge/
etesync-server    | [07/Nov/2020 20:06:47] "POST /api/v1/authentication/login_challenge/ HTTP/1.1" 401 50
etesync-server    | Unauthorized: /api/v1/authentication/login_challenge/
etesync-server    | [07/Nov/2020 20:06:51] "POST /api/v1/authentication/login_challenge/ HTTP/1.1" 401 50
etesync-server    | Unauthorized: /api/v1/authentication/login_challenge/
etesync-server    | [07/Nov/2020 20:07:28] "POST /api/v1/authentication/login_challenge/ HTTP/1.1" 401 50
plague-doctor commented 3 years ago

how can I turn on more verbose logging?

tasn commented 3 years ago

This long doesn't make sense tbh. login_challenge shouldn't return 401... Is this the log when doing signup?? Because there's no call to the signup endpoint in this log...

The best way to have good information on what's going on is use a tool like mitmproxy. For example, this is how i run it locally: mitmproxy -p 8034 --mode reverse:http://localhost:8033. Please use the latest mitmproxy if you can, as only that has native msgpack support (for nicer printing of the content).

plague-doctor commented 3 years ago

Sorry @tasn had no chance to play with mitmproxy, but have gone through the whole process again. Still no luck, but maybe you could check my configs? Anyway, this is the etesync-server logs (the whole process, from creating a new user, to attempt to login):

etesync-server    | Operations to perform:
etesync-server    |   Apply all migrations: admin, auth, contenttypes, django_etebase, myauth, sessions, token_auth
etesync-server    | Running migrations:
etesync-server    |   Applying contenttypes.0001_initial... OK
etesync-server    |   Applying contenttypes.0002_remove_content_type_name... OK
etesync-server    |   Applying auth.0001_initial... OK
etesync-server    |   Applying auth.0002_alter_permission_name_max_length... OK
etesync-server    |   Applying auth.0003_alter_user_email_max_length... OK
etesync-server    |   Applying auth.0004_alter_user_username_opts... OK
etesync-server    |   Applying auth.0005_alter_user_last_login_null... OK
etesync-server    |   Applying auth.0006_require_contenttypes_0002... OK
etesync-server    |   Applying auth.0007_alter_validators_add_error_messages... OK
etesync-server    |   Applying auth.0008_alter_user_username_max_length... OK
etesync-server    |   Applying auth.0009_alter_user_last_name_max_length... OK
etesync-server    |   Applying auth.0010_alter_group_name_max_length... OK
etesync-server    |   Applying auth.0011_update_proxy_permissions... OK
etesync-server    |   Applying myauth.0001_initial... OK
etesync-server    |   Applying admin.0001_initial... OK
etesync-server    |   Applying admin.0002_logentry_remove_auto_add... OK
etesync-server    |   Applying admin.0003_logentry_add_action_flag_choices... OK
etesync-server    |   Applying auth.0012_alter_user_first_name_max_length... OK
etesync-server    |   Applying django_etebase.0001_initial... OK
etesync-server    |   Applying django_etebase.0002_userinfo... OK
etesync-server    |   Applying django_etebase.0003_collectioninvitation... OK
etesync-server    |   Applying django_etebase.0004_collectioninvitation_version... OK
etesync-server    |   Applying django_etebase.0005_auto_20200526_1021... OK
etesync-server    |   Applying django_etebase.0006_auto_20200526_1040... OK
etesync-server    |   Applying django_etebase.0007_auto_20200526_1336... OK
etesync-server    |   Applying django_etebase.0008_auto_20200526_1535... OK
etesync-server    |   Applying django_etebase.0009_auto_20200526_1535... OK
etesync-server    |   Applying django_etebase.0010_auto_20200526_1539... OK
etesync-server    |   Applying django_etebase.0011_collectionmember_stoken... OK
etesync-server    |   Applying django_etebase.0012_auto_20200527_0743... OK
etesync-server    |   Applying django_etebase.0013_collectionmemberremoved... OK
etesync-server    |   Applying django_etebase.0014_auto_20200602_1558... OK
etesync-server    |   Applying django_etebase.0015_collectionitemrevision_salt... OK
etesync-server    |   Applying django_etebase.0016_auto_20200623_0820... OK
etesync-server    |   Applying django_etebase.0017_auto_20200623_0958... OK
etesync-server    |   Applying django_etebase.0018_auto_20200624_0748... OK
etesync-server    |   Applying django_etebase.0019_auto_20200626_0748... OK
etesync-server    |   Applying django_etebase.0020_remove_collectionitemrevision_salt... OK
etesync-server    |   Applying django_etebase.0021_auto_20200626_0913... OK
etesync-server    |   Applying django_etebase.0022_auto_20200804_1059... OK
etesync-server    |   Applying django_etebase.0023_collectionitemchunk_collection... OK
etesync-server    |   Applying django_etebase.0024_auto_20200804_1209... OK
etesync-server    |   Applying django_etebase.0025_auto_20200804_1216... OK
etesync-server    |   Applying django_etebase.0026_auto_20200907_0752... OK
etesync-server    |   Applying django_etebase.0027_auto_20200907_0752... OK
etesync-server    |   Applying django_etebase.0028_auto_20200907_0754... OK
etesync-server    |   Applying django_etebase.0029_auto_20200907_0801... OK
etesync-server    |   Applying django_etebase.0030_auto_20200922_0832... OK
etesync-server    |   Applying django_etebase.0031_auto_20201013_1336... OK
etesync-server    |   Applying django_etebase.0032_auto_20201013_1409... OK
etesync-server    |   Applying myauth.0002_auto_20200515_0801... OK
etesync-server    |   Applying sessions.0001_initial... OK
etesync-server    |   Applying token_auth.0001_initial... OK
etesync-server    | 
etesync-server    | 0 static files copied to '/var/www/static', 165 unmodified.
etesync-server    | Watching for file changes with StatReloader
etesync-server    | [08/Nov/2020 08:05:02] "GET /admin/ HTTP/1.1" 302 0
etesync-server    | [08/Nov/2020 08:05:02] "GET /admin/login/?next=/admin/ HTTP/1.1" 200 2198
etesync-server    | [08/Nov/2020 08:05:02] "GET /static/admin/css/base.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:05:02] "GET /static/admin/css/nav_sidebar.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:05:02] "GET /static/admin/css/responsive.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:05:02] "GET /static/admin/css/login.css HTTP/1.1" 200 1185
etesync-server    | [08/Nov/2020 08:05:02] "GET /static/admin/js/nav_sidebar.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:05:02] "GET /static/admin/css/fonts.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:05:02] "GET /static/admin/fonts/Roboto-Regular-webfont.woff HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:05:02] "GET /static/admin/fonts/Roboto-Light-webfont.woff HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:05:18] "POST /admin/login/?next=/admin/ HTTP/1.1" 200 2359
etesync-server    | [08/Nov/2020 08:05:18] "GET /static/admin/fonts/Roboto-Bold-webfont.woff HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:11] "POST /admin/login/?next=/admin/ HTTP/1.1" 302 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /admin/ HTTP/1.1" 200 3544
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/css/base.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/css/nav_sidebar.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/css/dashboard.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/js/nav_sidebar.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/css/responsive.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/css/fonts.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/img/icon-addlink.svg HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/fonts/Roboto-Bold-webfont.woff HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/fonts/Roboto-Regular-webfont.woff HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/img/icon-changelink.svg HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:12] "GET /static/admin/fonts/Roboto-Light-webfont.woff HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /admin/myauth/user/add/ HTTP/1.1" 200 7067
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/css/forms.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/js/vendor/jquery/jquery.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/js/admin/RelatedObjectLookups.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/js/core.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/js/jquery.init.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/css/widgets.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/js/actions.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/js/prepopulate.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/js/urlify.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/js/prepopulate_init.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/js/vendor/xregexp/xregexp.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:17] "GET /admin/jsi18n/ HTTP/1.1" 200 3187
etesync-server    | [08/Nov/2020 08:07:17] "GET /static/admin/js/change_form.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "POST /admin/myauth/user/add/ HTTP/1.1" 302 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /admin/myauth/user/2/change/ HTTP/1.1" 200 19904
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/js/calendar.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/js/SelectFilter2.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/js/SelectBox.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/js/admin/DateTimeShortcuts.js HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /admin/jsi18n/ HTTP/1.1" 200 3187
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/img/icon-yes.svg HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/img/icon-unknown.svg HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/img/icon-clock.svg HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/img/selector-icons.svg HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/img/icon-calendar.svg HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/img/icon-unknown-alt.svg HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:28] "GET /static/admin/img/search.svg HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:36] "POST /admin/myauth/user/2/change/ HTTP/1.1" 302 0
etesync-server    | [08/Nov/2020 08:07:36] "GET /admin/myauth/user/ HTTP/1.1" 200 8564
etesync-server    | [08/Nov/2020 08:07:36] "GET /static/admin/css/changelists.css HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:36] "GET /admin/jsi18n/ HTTP/1.1" 200 3187
etesync-server    | [08/Nov/2020 08:07:36] "GET /static/admin/img/tooltag-add.svg HTTP/1.1" 304 0
etesync-server    | [08/Nov/2020 08:07:36] "GET /static/admin/img/sorting-icons.svg HTTP/1.1" 304 0
etesync-server    | Unauthorized: /api/v1/authentication/login_challenge/
etesync-server    | [08/Nov/2020 08:07:45] "POST /api/v1/authentication/login_challenge/ HTTP/1.1" 401 50

docker-compose.yaml:

---
# EteSync

version: '2'

services:
  etesync-server:
    image: plaguedr/etesync-server:latest
    container_name: etesync-server
    hostname: etesync-server
    networks:
      - proxy
      - etesync
    volumes:
      - /mnt/docker/docker-volumes/etebase/data:/data
      - /mnt/docker/docker-volumes/etebase/web:/var/www
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.etesync-server.rule=Host(`etesync-server.domain.my`)"
      - "traefik.http.routers.etesync-server.tls.certresolver=letsencrypt"
      - "traefik.http.routers.etesync-server.entrypoints=https"
      - "traefik.http.routers.etesync-server.tls=true"
      - "traefik.http.services.etesync-server.loadbalancer.server.port=8000"
    environment:
      SUPER_USER: admin
      SERVER: standalone
      DJANGO_LC: en-au
      TZ: Australia/Melbourne
      ETEBASE_ALLOWED_HOSTS: "*"
    restart: always

  etesync-web:
    image: plaguedr/etesync-web:latest
    container_name: etesync-web
    hostname: etesync-web
    networks:
      - proxy
      - etesync
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.etesync-web.rule=Host(`etesync-web.domain.my`)"
      - "traefik.http.routers.etesync-web.tls.certresolver=letsencrypt"
      - "traefik.http.routers.etesync-web.entrypoints=https"
      - "traefik.http.routers.etesync-web.tls=true"
      - "traefik.http.services.etesync-web.loadbalancer.server.port=8000"
    environment:
      ETESYNC_URL: "https://etesync-server.domain.my"
    depends_on:
      - etesync-server
    restart: always

networks:
  proxy:
    external: true
  etesync:
    driver: bridge

Docker entrypoint.sh:

#!/bin/sh

set -e

if [ -n "$1" ]; then
    exec "$@"
    exit 0
fi

if [ ! -e "$ETEBASE_DB_PATH" ]; then
    # first run
    mkdir -p "$DJANGO_STATIC_ROOT" "$DJANGO_MEDIA_ROOT" "$DJANGO_TEMPLATES_ROOT"
    "$ETESYNC_PATH"/manage.py migrate
    chown -R "$UUID":"$GID" "$ETEBASE_DATA_PATH" "$DJANGO_STATIC_ROOT" "$DJANGO_MEDIA_ROOT" "$DJANGO_TEMPLATES_ROOT"

    if [ "$SUPER_USER" ] && [ "$SUPER_PASS" ]; then
        echo "from django.contrib.auth.models import User; User.objects.create_superuser('$SUPER_USER' , '$SUPER_EMAIL', '$SUPER_PASS')" | python manage.py shell
    fi
fi

if [ "$SERVER" = 'uwsgi' ]; then
    /usr/local/bin/uwsgi --ini etesync.ini
elif [ "$SERVER" = 'uwsgi-http' ]; then
    /usr/local/bin/uwsgi --ini etesync.ini:http
else
    "$ETESYNC_PATH"/manage.py collectstatic --no-input
    "$ETESYNC_PATH"/manage.py runserver 0.0.0.0:"$DJANGO_PORT"
fi

settings.py:

"""
Django settings for etebase_server project.

Generated by 'django-admin startproject' using Django 3.0.3.

For more information on this file, see
https://docs.djangoproject.com/en/3.0/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.0/ref/settings/
"""

import os
import configparser
from .utils import get_secret_from_file

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DATA_DIR = os.environ.get('ETEBASE_DATA_PATH', '/data')

AUTH_USER_MODEL = 'myauth.User'

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
# See secret.py for how this is generated; uses a file 'secret.txt' in the root
# directory
SECRET_FILE = os.path.join(DATA_DIR, "secret.txt")

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = [os.environ.get('ETEBASE_ALLOWED_HOSTS')]

# Database
# https://docs.djangoproject.com/en/2.0/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.environ.get('ETEBASE_DB_PATH',
                               os.path.join(BASE_DIR, 'db.sqlite3')),
    }
}

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'corsheaders',
    'rest_framework',
    'myauth.apps.MyauthConfig',
    'django_etebase.apps.DjangoEtebaseConfig',
    'django_etebase.token_auth.apps.TokenAuthConfig',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'corsheaders.middleware.CorsMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'etebase_server.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            os.path.join(BASE_DIR, 'templates')
        ],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'etebase_server.wsgi.application'

# Password validation
# https://docs.djangoproject.com/en/3.0/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

# Internationalization
# https://docs.djangoproject.com/en/3.0/topics/i18n/

LANGUAGE_CODE = 'en-us'
TIME_ZONE = os.environ.get('TZ', 'UTC')
USE_I18N = True
USE_L10N = True
USE_TZ = True

# Cors
CORS_ORIGIN_ALLOW_ALL = True

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.0/howto/static-files/

STATIC_URL = '/static/'
STATIC_ROOT = os.environ.get('DJANGO_STATIC_ROOT', os.path.join(BASE_DIR, 'static'))

MEDIA_ROOT = os.environ.get('DJANGO_MEDIA_ROOT', os.path.join(BASE_DIR, 'media'))
MEDIA_URL = '/user-media/'

# Define where to find configuration files
config_locations = ['etebase-server.ini', '/etc/etebase-server/etebase-server.ini']
# Use config file if present
if any(os.path.isfile(x) for x in config_locations):
    config = configparser.ConfigParser()
    config.read(config_locations)

    section = config['global']

    SECRET_FILE   = section.get('secret_file', SECRET_FILE)
    STATIC_ROOT   = section.get('static_root', STATIC_ROOT)
    STATIC_URL    = section.get('static_url', STATIC_URL)
    MEDIA_ROOT    = section.get('media_root', MEDIA_ROOT)
    MEDIA_URL     = section.get('media_url', MEDIA_URL)
    LANGUAGE_CODE = section.get('language_code', LANGUAGE_CODE)
    TIME_ZONE     = section.get('time_zone', TIME_ZONE)
    DEBUG         = section.getboolean('debug', DEBUG)

    if 'allowed_hosts' in config:
        ALLOWED_HOSTS = [y for x, y in config.items('allowed_hosts')]

    if 'database' in config:
        DATABASES = { 'default': { x.upper(): y for x, y in config.items('database') } }

ETEBASE_API_PERMISSIONS = ('rest_framework.permissions.IsAuthenticated', )
ETEBASE_API_AUTHENTICATORS = ('django_etebase.token_auth.authentication.TokenAuthentication',
                              'rest_framework.authentication.SessionAuthentication')
ETEBASE_CREATE_USER_FUNC = 'django_etebase.utils.create_user_blocked'

# Make an `etebase_server_settings` module available to override settings.
try:
    from etebase_server_settings import *
except ImportError:
    pass

if 'SECRET_KEY' not in locals():
    SECRET_KEY = get_secret_from_file(SECRET_FILE)

uwsgi.ini:

[uwsgi]
ini = :base
protocol = uwsgi
socket = 0.0.0.0:$(DJANGO_PORT)

[http]
ini = :base
protocol = http
http = 0.0.0.0:$(DJANGO_PORT)

[base]
master = True
workers = 2
chdir = $(ETESYNC_PATH)
wsgi-file = $(ETESYNC_PATH)/etebase_server/wsgi.py
uid = $(PGID)
gid = $(PGID)
static-map = /static=/var/www/static
static-expires = /* 7776000
harakiri = 60
vacuum = True
socket = :$(DJANGO_PORT)
pidfile = /tmp/etebase_server.pid
max-requests = 5000
buffer-size = 25000
post-buffering = 8192
enable-threads = True
die-on-term = True
tasn commented 3 years ago

Ah, sorry, I checked again, 401 is correct for login_challenge, what's incorrect is that it should be followed by a signup. Could you please try using https://pim.etesync.com (even on a test account)? I suspect there's something up with your local web client.

plague-doctor commented 3 years ago

That is a very strange thing indeed... I have stopped all my services I am hosting, leaving only three: etesync-server, etesync-web and traefik. And guess what? It started to work! I can login to etesync-web... I have got a strong suspicion traefik was doing something dogdy...

But now I can see a peculiar behavior. Etesync-web works in chromium and ungoogled-chromium, but it does not work from FireFox. I am unable to login using FireFox... This is what I get: etesync-ff

Thnaks you @tasn for all your help. Looks like I need to drill deeper into traefik - this is strange indeed.

plague-doctor commented 3 years ago

Hmmm.... Have I found a bug?

Something went wrong!
invalid key length
TypeError: invalid key length
    at v (https://etesync-web.domain.my/static/js/2.d3ee8ba9.chunk.js?__WB_REVISION__=e14987bd47054fd0af4a:2:134054)
    at Object.Qe [as crypto_kdf_derive_from_key] (https://etesync-web.domain.my/static/js/2.d3ee8ba9.chunk.js?__WB_REVISION__=e14987bd47054fd0af4a:2:161211)
    at new e (https://etesync-web.domain.my/static/js/2.d3ee8ba9.chunk.js?__WB_REVISION__=e14987bd47054fd0af4a:2:688852)
    at n.<anonymous> (https://etesync-web.domain.my/static/js/2.d3ee8ba9.chunk.js?__WB_REVISION__=e14987bd47054fd0af4a:2:1608438)
    at new n (https://etesync-web.domain.my/static/js/2.d3ee8ba9.chunk.js?__WB_REVISION__=e14987bd47054fd0af4a:2:695209)
    at Ae (https://etesync-web.domain.my/static/js/2.d3ee8ba9.chunk.js?__WB_REVISION__=e14987bd47054fd0af4a:2:696973)
    at e.value (https://etesync-web.domain.my/static/js/2.d3ee8ba9.chunk.js?__WB_REVISION__=e14987bd47054fd0af4a:2:725979)
    at e.get (https://etesync-web.domain.my/static/js/2.d3ee8ba9.chunk.js?__WB_REVISION__=e14987bd47054fd0af4a:2:739420)
    at Pi (https://etesync-web.domain.my/static/js/main.50404d52.chunk.js?__WB_REVISION__=81acd70a9c2aa44094ca:1:263830)
    at Wo (https://etesync-web.domain.my/static/js/2.d3ee8ba9.chunk.js?__WB_REVISION__=e14987bd47054fd0af4a:2:1510272)
tasn commented 3 years ago

What's going on, if I had to guess: you have some CSP or an add-on on firefox blocking the WASM code from running so it's falling back to the VERY slow asm.js code. Try removing some of your custom stuff, get it to work, and then enable them back again seeing what breaks.

Anyhow, I'm closing this ticket as the main issue has been fixed.