Josue-T commented 4 months ago

Matrix V2 Implementation

This is more a tracking page and discussion about Matrix v2 implementation than a pull request.

The target here is to implement all component of matrix V2 so:

[x] Matrix sliding sync proxy: https://github.com/matrix-org/sliding-sync
[ ] Element call: https://github.com/vector-im/element-call
[ ] Matrix authentication service: https://github.com/matrix-org/matrix-authentication-service

The idea here is to provide for user how want a beta version of synapse which will implement theses new feature. Note that this branch will be updated in same time than the testing so user can easily install the branch and get the update as same as than for testing.

frequently asked questions

Why not merging this and making this available to more people ?

Because some component are still really experimental on matrix side. On sliding proxy projet side we have currently theses issues:

This project are built to run only with new libc version which is not available on debian bullseye. A chroot was made to bypass this issue but it's not a clean way to implement this. The best solution about this is to wait for debian bookworm on yunohost.
The build of this project are available on for arm64 and amd64 architecture so it's really restricted.

How can I use this branch and experiment theses feature ?

As same as for testing you can install the app with this command:

sudo yunohost app install https://github.com/YunoHost-Apps/synapse_ynh/tree/matrix_v2

And you can also update the app with this command:

sudo yunohost app upgrade synapse -u https://github.com/YunoHost-Apps/synapse_ynh/tree/matrix_v2

Can I install this without breaking my instance ?

Short answer: the stability of this branch will be same than testing

Long answer: As same as for testing we try to avoid any issue with these branch and try to keep it stable. But it still be a community project and we can't guarantee the stability of the code. Note also that the stability depends of the upstream code.

When will this be merged into master ?

It depends of the development on package side but also of the upstream on matrix/element side so for now it's difficult to say. In all case the idea is not to merge before all component are ready and tested on the package.

Why adding element-call into this package as there are already a package with this

The idea here is to provide a package with all main feature of matrix configured directly into synapse so there is no more setup to do.

Can I switch back to master or testing branch after doing an upgrade from this branch ?

Nothing was designed for this, so there are a huge risk to break something or that something don't work any-more if you do this. But note that this branch in long term will be merged into testing than master so you won't be blocked if you move to this branch.

TODO

[x] Finish Sliding sync proxy #440
[ ] Implement Element call
[ ] Implement Matrix authentication service
[ ] Drop Debian bullseye support and clean-up sliding sync proxy implementation (when Yunohost will support officially debian bookworm).

lapineige commented 3 months ago

I want to congratulate @Josue-T for the huge recent work on both these improvements and work on packaging v2 #426 and various fixes 🎉 👏 Thanks also to all people contributing to this :)

Thatoo commented 3 months ago

Thank you so much @Josue-T for your incredible amount of work these last days.

centralscrutinizer21 commented 3 months ago

Adding my thanks to @Josue-T as well. I am going to test this branch and face the wrath of the 3 friends I convinced to get on my matrix instance 😄

centralscrutinizer21 commented 3 months ago

Installation went fine, upgrading not so much. I still can't get to use Element x maybe something related to this warning? https://paste.yunohost.org/agadaqileg Also, do I need to open ports 8008 8009 ?

Josue-T commented 3 months ago

Normally all port are already open. But well what happen wrong. The upgrade process or the service was not working after the upgrade ?

centralscrutinizer21 commented 3 months ago

The service is running fine. Of course the two bridges (whatsapp & telegram) are not working anymore. As I said I tried to login with Element X . Ii recognise the server as valid, but then it cannot reach it. From the logs it seems that there could be some syntax erro maybe?

Josue-T commented 3 months ago

Do you have some log of theses bridges ?

centralscrutinizer21 commented 3 months ago

Sorry, my bad. I had a second application on the same domain, and that screwed up the installation. Now it's working. Where do I find the procedure to move synapse to a different domain keeping the same user handle?

Josue-T commented 3 months ago

I created an issue about this here: https://github.com/YunoHost-Apps/synapse_ynh/issues/443

centralscrutinizer21 commented 3 months ago

Yes, thank you I found it and succesfully moved synapse to a different domain. I am just not very proficient with the syntax of /etc/ssowat/conf.json.persistent can you show me how it should look?

Josue-T commented 3 months ago

About change url and domain constraints issue please post your message on #443

nathanael-h commented 3 months ago

Hello, I successfully upgraded to this branch. The Sliding Sync proxy's service is enabled and running well. As far as I see Synapse and the slinding sync proxy are installed on the same domain. Unfortunately I can't make Element X on Android connect to my server.

My matrix server is running at matrix.domain.fr and my main domain is domain.fr the .well-known are advertised here :

curl https://domain.fr/.well-known/matrix/client 
{
        "m.homeserver": { "base_url": "https://matrix.domain.fr" },
        "org.matrix.msc3575.proxy": { "url": "https://matrix.domain.fr"},
        "im.vector.riot.jitsi": {"preferredDomain": "jitsi.riot.im"},
        "im.vector.riot.e2ee": {"default": true }
    }%

When I try to connect Element X to domain.fr I have a "Nous n'avons pas pu atteindre ce serveur d'accueil. Vérifiez que vous avez correctement saisi l'URL du serveur d'accueil. Si l'URL est correcte contactez votre administrateur... etc." I see this entries in logs nignx :

/var/log/nginx/domain.fr-access.log:90.66.163.140        - - [18/Mar/2024:18:17:03 +0100] "GET /.well-known/matrix/client HTTP/2.0" 200 273 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
/var/log/nginx/matrix.domain.fr-access.log:90.66.163.140 - - [18/Mar/2024:18:17:03 +0100] "GET /_matrix/client/versions HTTP/2.0" 200 929 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
/var/log/nginx/matrix.domain.fr-access.log:90.66.163.140 - - [18/Mar/2024:18:17:03 +0100] "GET /_matrix/client/v3/login HTTP/2.0" 200 170 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"

I'd like to help you with this pull request and adding the sliding sync proxy, by the way thank you for all you've already done.

Josue-T commented 3 months ago

Hello, I think you should define the server as matrix.domain.fr and not domain.fr ? Normally this should works.

nathanael-h commented 3 months ago

I tried to login with matrix.domain.fr URL in Element X, but it says "Error this homeserver does not support sliding sync".

In nginx logs I see this :

root@domain:/etc/nginx/conf.d# grep "Element X" /var/log/nginx/*.log
/var/log/nginx/domain.fr-access.log:90.66.163.140 - - [19/Mar/2024:10:05:55 +0100] "GET /yunohost/sso/?r=aHR0cHM6Ly9tYXRyaXguaXNpZG9ydXxxxxxxxxxxx4vbWF0cml4L2NsaWVudA== HTTP/2.0" 200 3077 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
/var/log/nginx/domain.fr-access.log:90.66.163.140 - - [19/Mar/2024:10:06:05 +0100] "GET /yunohost/sso/?r=aHR0cHM6Ly9tYXRyaXguaXNpZG9ydXxxxxxxxxxxx4vbWF0cml4L2NsaWVudA== HTTP/2.0" 200 3077 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
/var/log/nginx/domain.fr-access.log:90.66.163.140 - - [19/Mar/2024:10:06:08 +0100] "GET /yunohost/sso/?r=aHR0cHM6Ly9tYXRyaXguaXNpZG9ydXxxxxxxxxxxx4vbWF0cml4L2NsaWVudA== HTTP/2.0" 200 3077 "https://matrix.domain.fr/.well-known/matrix/client" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"

/var/log/nginx/matrix.domain.fr-access.log:90.66.163.140 - - [19/Mar/2024:10:05:55 +0100] "GET /.well-known/matrix/client HTTP/2.0" 302 138 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
/var/log/nginx/matrix.domain.fr-access.log:90.66.163.140 - - [19/Mar/2024:10:06:05 +0100] "GET /.well-known/matrix/client HTTP/2.0" 302 138 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
/var/log/nginx/matrix.domain.fr-access.log:90.66.163.140 - - [19/Mar/2024:10:06:05 +0100] "GET /.well-known/matrix/client HTTP/2.0" 302 138 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
/var/log/nginx/matrix.domain.fr-access.log:90.66.163.140 - - [19/Mar/2024:10:06:07 +0100] "GET /.well-known/matrix/client HTTP/2.0" 302 138 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
/var/log/nginx/matrix.domain.fr-access.log:90.66.163.140 - - [19/Mar/2024:10:06:08 +0100] "GET /_matrix/client/versions HTTP/2.0" 200 929 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
/var/log/nginx/matrix.domain.fr-access.log:90.66.163.140 - - [19/Mar/2024:10:06:08 +0100] "GET /_matrix/client/versions HTTP/2.0" 200 929 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
/var/log/nginx/matrix.domain.fr-access.log:90.66.163.140 - - [19/Mar/2024:10:06:08 +0100] "GET /_matrix/client/v3/login HTTP/2.0" 200 170 "-" "Element X/0.4.5 (Fairphone FP3; Android 12; e_FP3-user 12 SQ3A.220705.004 eng.root.20240220.205127 dev-keys; Sdk TODO)"
root@domain:/etc/nginx/conf.d#

And if I curl https://matrix.domain.fr/.well-known/matrix/client I have a 302 to SSO ; indeed the .well-know/matrix/client is on the main yunohost domain, i.e. domain.fr... It looks like the Element X app is first calling the .well-know/matrix/client path on the URL I set as the homeserver URL, then nginx says 302 go to SSO, then it fails.

I see no entry in sliding-sync service's journal.

Josue-T commented 3 months ago

Hello,

Did some test, the issue is linked that Element X don't support CAS and if password authentication is disabled it don't works. By d42f646 it should work now.

nathanael-h commented 3 months ago

I confirm it works: I could log in Element X, indeed with username and password!

Bugsbane commented 3 months ago

Is it possible to install this on a separate testing subdomain to stable Synapse? I understand I'd need to use a separate test user at the testing subdomain.

Josue-T commented 2 months ago

Is it possible to install this on a separate testing subdomain to stable Synapse? I understand I'd need to use a separate test user at the testing subdomain.

Well, you can install a seconds independent instance of synapse which use this version, but you can't use this branch with the same instance that you don't want to support matrix v2. But well the idea is to keep this branch stable and if there are a development which is done for this, it will be done on a specific branch until it's stable.

Josue-T commented 2 months ago

!testme

yunohost-bot commented 2 months ago

May the CI gods be with you!

yunohost-bot commented 2 months ago

:worm:

centralscrutinizer21 commented 2 months ago

Hello @Josue-T could you patch your draft with this https://github.com/YunoHost-Apps/synapse_ynh/commit/1ef97c77abc26e9a7d539ec4eaef181542163dbc ? I am trying to test the functionality of the new go bridges on your branch

nathanael-h commented 2 months ago

In case Josué would not be available, you could still apply this patch manually after installing this branch

diff --git a/sources/update_synapse_for_appservice.sh b/sources/update_synapse_for_appservice.sh
index bf6979c..5abacfe 100644
--- a/sources/update_synapse_for_appservice.sh
+++ b/sources/update_synapse_for_appservice.sh
@@ -18,8 +18,8 @@ for f in $(ls /etc/matrix-$app/app-service/); do
 done

 # Set permissions
-chown matrix-$app $service_config_file
-chown matrix-$app /etc/matrix-$app/app-service/*
+chown $app $service_config_file
+chown $app /etc/matrix-$app/app-service/*
 chmod 600 $service_config_file
 chmod 600 /etc/matrix-$app/app-service/*

And then you could test install/upgrade of bridges.

centralscrutinizer21 commented 2 months ago

In case Josué would not be available, you could still apply this patch manually after installing this branch

diff --git a/sources/update_synapse_for_appservice.sh b/sources/update_synapse_for_appservice.sh
index bf6979c..5abacfe 100644
--- a/sources/update_synapse_for_appservice.sh
+++ b/sources/update_synapse_for_appservice.sh
@@ -18,8 +18,8 @@ for f in $(ls /etc/matrix-$app/app-service/); do
 done

 # Set permissions
-chown matrix-$app $service_config_file
-chown matrix-$app /etc/matrix-$app/app-service/*
+chown $app $service_config_file
+chown $app /etc/matrix-$app/app-service/*
 chmod 600 $service_config_file
 chmod 600 /etc/matrix-$app/app-service/*

And then you could test install/upgrade of bridges.

I'm sorry I am not very good with cli commands.... what commands should I type in terminal? I have ssh access to my server

centralscrutinizer21 commented 2 months ago

Ok I figured it out.... I needed to modify the file /opt/yunohost/matrix-synapse/update_synapse_for_appservice.sh

Josue-T commented 2 months ago

Hello @Josue-T could you patch your draft with this 1ef97c7 ? I am trying to test the functionality of the new go bridges on your branch

Done :wink:

centralscrutinizer21 commented 2 months ago

For the records, Succesfully installed synapse from this draft, matrix-telegram and matrix-whatsapp from their respective master branch and matrix-signal from ci-auto-update-0.5.1 branch. Everything is working fine!

nathanael-h commented 2 months ago

This branch stills works well, thanks.

nathanael-h commented 3 weeks ago

Hello, could you update the sliding sync proxy? Here is an MR https://github.com/YunoHost-Apps/synapse_ynh/pull/465

Thatoo commented 2 weeks ago

sounds very promising all that :-)

Josue-T commented 1 week ago

!testme

yunohost-bot commented 1 week ago

:stuck_out_tongue_winking_eye:

yunohost-bot commented 1 week ago

Living in the future, are we?

xabirequejo commented 3 days ago

Since FluffyChat doesn't deliver notifications for other than the 1st account, I'd like to use Element X. So, those who have updated from Synapse Matrix v1 to v2:

have you had any issue?
Are bridges still working?
Did you need to reconnect them again?
And is Synapse Matrix v2 compatible with Element (the old one, my iPad can't install Element X)?

Josue-T commented 2 days ago

Look like that we will have a native implementation instead of the a proxy in middle term: cf https://github.com/element-hq/synapse/pull/17187

YunoHost-Apps / synapse_ynh

Matrix v2 #439

Matrix V2 Implementation

frequently asked questions

Why not merging this and making this available to more people ?

How can I use this branch and experiment theses feature ?

Can I install this without breaking my instance ?

When will this be merged into master ?

Why adding element-call into this package as there are already a package with this

Can I switch back to master or testing branch after doing an upgrade from this branch ?

TODO