search

element-hq / roadmap

Element public roadmap
78 stars 5 forks source link

Gitter feature parity overview issue #26

Open MadLittleMods opened 3 years ago

MadLittleMods commented 3 years ago

In the long-run we want Element to take-over for Gitter because we don't want to support both platforms. In order to have Element be a suitable successor, we want to make sure all of the feature functionality people are used to on Gitter is available in Element. If you want to read more context behind this, the Matrix blog post has great detail. All of the Gitter content will be available on the gitter.im Matrix homeserver to live on.

Feel free to edit this list with more features, details, and links(MSC's, etc) as you think of them. We can split this out individual issues once this list has a few more iterations.

Gitter features needed in Element

Extras

Sunset

After Gitter shuts down, we want the experience of accessing your same Gitter content on Matrix just as seamless.

Previous after Gitter sunset idea: push people to their own homeserver - [ ] Sign in with GitLab, GitHub, Twitter -> automatically bootstrap their `matrix.org` account and be signed in - This way, we don’t have to pay for beefing up the `gitter.im` homeserver with people trying to join big rooms - We want the experience to be easy for user with no knowledge of Matrix. They should just need to sign in with GitLab/GitHub like they did with Gitter and we will handle the Matrix user creation and plop them in. This will also add admin permissions for people in the rooms they were admins in. - People may sign in and get the automatic Matrix.org account created but want to access their rooms from another Matrix account. We should also have a migration utility to be able `/migrate_room_membership_to @xxx` and have it invite their other user to the same rooms. - EMS has a generic MXID migration tool we can point people to, https://ems.element.io/tools/matrix-migration - Instead of pushing everyone to `matrix.org`, maybe suggest EMS for your own homeserver or maybe P2P if that's mature in time. The one problem with this flow is it would probably be super cumbersome and clumsy for someone not familiar with Matrix, e.g. "Why do I have to create a server if I just want to sign into Gitter?"

Master migration rollout plan

  1. [x] Import all messages from Gitter -> Matrix, https://gitlab.com/gitterHQ/webapp/-/issues/2609
  2. [x] Sync membership in all live Matrix rooms, https://gitlab.com/gitterHQ/webapp/-/issues/2857
  3. [x] Sync membership in all historical Matrix rooms, https://gitlab.com/gitterHQ/webapp/-/issues/2857
  4. [x] Sync admin permissions to all live and historical Matrix rooms
  5. [x] Pre-announcement cutover blog post, https://docs.google.com/document/d/1WNuQ3s8dNn7uxdkG515Hnl07RhnCrXC7jmUP1r6czJA/edit# -> https://blog.gitter.im/2023/01/16/gitter-is-going-fully-native-matrix-in-feb-2023/
  6. [x] Link pre-announcement from announcement section of the left-menu
  7. [x] Ensure live room points back to historical room via MSC3946,
  8. [x] Inject social provider information into Synapse, https://github.com/vector-im/sre-internal/issues/1754
  9. [x] Setup app.gitter.im with the Gitter branded Element instance, https://github.com/matrix-org/matrix-hosted/issues/6824
  10. [x] Setup /.well-known/matrix/client so people can enter gitter.im on the Element sign-in page
  11. [x] Update Matrix appservice registration to not have exclusive control over the users (make sure updated on EMS), https://github.com/matrix-org/matrix-hosted/issues/6979
  12. [x] Remove custom Synapse patches used to optimize during the import process, https://github.com/matrix-org/matrix-hosted/issues/7144
  13. [x] Deploy latest Element which includes some updates for Gitter, https://github.com/matrix-org/matrix-hosted/issues/7148
  14. [x] Allow anonymous preview of rooms on app.gitter.im, https://github.com/matrix-org/matrix-hosted/issues/7150
  15. [x] Enable social login on app.gitter.im
  16. [x] Blog post explaining the Gitter -> Element/Matrix transition, https://docs.google.com/document/d/1s05BHAz-EFKktfBZ3BD2j5ojR2xIwDR13t_6XiNdI_M/edit#
    1. [x] FAQ document we can link wherever
    2. -> https://blog.gitter.im/2023/02/13/gitter-has-fully-migrated-to-matrix/

--- Cutover on 2023-02-13 ---

  1. [x] Update Gitter homepage to direct people to app.gitter.im so all new people go in via Matrix
  2. [x] Disallow sign-in to Gitter itself. Update sign-in page to point to app.gitter.im to better direct Gitter desktop users to the right place
  3. [x] Also need to stop people generating tokens from their already created OAuth apps. Probably cut-off token exchange endpoint.
  4. [x] Disallow sign-in on https://developer.gitter.im/ to stop people from generating new tokens
  5. [x] Again -> Inject social provider information into Synapse, https://github.com/vector-im/sre-internal/issues/1754
    • Resume dump from gitterUserId=63bc7a736da0373984af769a (small amount of overlap from our last dump)
    • Resume dump from gitterUserId=63e9a3486da0373984be49f7
  6. [x] Invalidate all Gitter access tokens so people get a 401 and are redirected back to the home page. People will then stumble back in via app.gitter.im
    • This will result in a confusing hard cut-off for the Gitter iOS and Android apps which have been deprecated since 2020-11-20
    • When we invalidate all tokens, this will also break people's scripts/bots. Might just have to rip off the band-aid on this one.
  7. [x] Setup redirects from old Gitter URL scheme to Element/Matrix location, https://github.com/vector-im/sre-internal/issues/1755, https://gitlab.com/gitterHQ/gitter.im/-/issues/9
  8. [x] Stop Gitter unread emails from going out, https://gitlab.com/gitterHQ/gitter-infrastructure/-/issues/261
  9. [x] Shutdown the IRC bridge, https://gitlab.com/gitterHQ/gitter-infrastructure/-/issues/262
  10. [x] Remove Android and iOS apps from the app stores, https://github.com/vector-im/sre-internal/issues/1854
  11. [x] Stop all public traffic to AWS Gitter infra
    • Be mindful that people still need to access the Gitter homepage in case we're still serving that from the Gitter infra

--- Cleanup ---

  1. [x] Archive Gitter Mongo data, https://github.com/vector-im/sre-internal/issues/1929
  2. [x] Shutdown AWS Gitter infra
  3. [x] Cancel and stop paying for associated services, https://github.com/vector-im/sre-internal/issues/1927
  4. [x] Archive projects, https://gitlab.com/gitterHQ
NickHu commented 2 years ago

I think it's a bit misleading to mark the Maths feature as done. For web it works, but I need help finishing https://github.com/vector-im/element-android/pull/2133, and as far as I'm aware nobody has started implementing it on iOS. My PR has stalled for over a year now, and I'm quite sure that it doesn't need much developer time to merge.

marclaporte commented 1 year ago

It's happening!

"So, after over 2 years, we’re pressing the button at last: we’re migrating all the existing content in the original centralised Gitter platform over into Gitter’s Matrix server, and we’re planning to replace the old Gitter app with a Gitter-customised version of Element during the week of Feb. 6th 2023."

https://blog.gitter.im/2023/01/16/gitter-is-going-fully-native-matrix-in-feb-2023/

csarven commented 1 year ago

Setup redirects to rewrite the old Gitter URL to Matrix room alias, https://github.com/vector-im/sre-internal/issues/1755

That URL is 404 for the public or at least I'm not authorized to read. Can the comment be updated to refer to the issue/PR addressing redirect or some information about its status?

MadLittleMods commented 1 year ago

@csarven The links are up-to-date but those are internal repositories that you won't have access to.

melvincarvalho commented 1 year ago

The sunset period is FAR too short. Please consider extending it.

timbl commented 1 year ago

So now I have the Element app on Mac, ... I see one room is in bridged from Gitter, but I don't see any other Gitter rooms. How to I open an arbitrary Gitter room in Element? How from Element can I see what Gitter rooms exist?

timbl commented 1 year ago

Any recommendations for (command line) matrix api help?

leonardehrenfried commented 1 year ago

@timbl You need to search the public room directory and select gitter.im as the target for the search.

Screenshot 2023-02-05 at 19 43 30
leonardehrenfried commented 1 year ago

BTW, all public rooms are bridged.

timbl commented 1 year ago
image

Searching Public Rooms for 'gitter' does not seem to give me gitter rooms .. and searching for "solidos" doesn't give me our solidos room

timbl commented 1 year ago

sorry that was 'gitter' not 'gitter.im'

leonardehrenfried commented 1 year ago

You need to select "gitter.im" from the drop-down below the search box. In your screenshot it says "Matrix rooms (matrix.org)" in mine the drop down is opened.

melvincarvalho commented 1 year ago

@leonardehrenfried thanks for the info. Trying this. It just says "joining". Is it too early?

image

leonardehrenfried commented 1 year ago

Don't really know. I was able to join the room with my matrix user successful. Maybe retry?

melvincarvalho commented 1 year ago

Retrying worked, thank you!

timbl commented 1 year ago

@leonardehrenfried It looks as though the gitter rooms like solid/solidos may all have been bridged to things like https://matrix.to/#/#solid_solidos:gitter.im -- is that right?

leonardehrenfried commented 1 year ago

Yes that's to be expected. Seems that room addresses can't contain slashes.

leonardehrenfried commented 1 year ago

The room address is #solid_solidos:gitter.im.

Are you aware that starting tomorrow the gitter backend will be converted to use the matrix protocol only?

MadLittleMods commented 1 year ago

Are you aware that starting tomorrow the gitter backend will be converted to use the matrix protocol only?

The migration date has moved to 2023-02-13 (next Monday). On this day, you will get access to sign-in to the gitter.im homeserver and will have access to the public and private rooms that you were previously using on Gitter directly (membership and admin permissions).

Public rooms are already available to join if that's all you need (this has been a thing for a couple years now)

jakirkham commented 1 year ago

First want to say thank you for all the hard work you are doing! Am really looking forward to using Matrix more generally (especially for things that are only available through Gitter atm like private rooms) 😄

The migration date has moved to 2023-02-13 (next Monday).

IIRC this was originally going to happen today. Curious why this was moved back a week (if you are able to share). Recognize people working on this are likely very busy. So don't want to distract. Just having a little more context would be helpful (maybe one sentence would be sufficient). Sorry again for the intrusion.

MadLittleMods commented 1 year ago

IIRC this was originally going to happen today. Curious why this was moved back a week (if you are able to share). Recognize people working on this are likely very busy. So don't want to distract. Just having a little more context would be helpful (maybe one sentence would be sufficient). Sorry again for the intrusion.

@jakirkham Mostly just prep work that collided with FOSDEM which happened this past weekend.

ekpyron commented 1 year ago

Just to let you know (I reckon what I'm seeing is related to this issue here): I take it you're just running https://gitlab.com/gitterHQ/webapp/-/issues/2857 ? This doesn't seem to scale all that well - I have a room with over 9000 gitter users (https://gitter.im/ethereum/solidity) that's now been unusable for over an hour due to the matrix side apparently not being able to handle the amount of users being joined to the room at once all that well (If I read the notification element gives me correctly, after an hour it has synced only half of the users). Not sure if me viewing the room via federation makes this worse and whether this will settle eventually - if it keeps taking several hours to properly open the room from now on, that'd be a problem :-).

MadLittleMods commented 1 year ago

@ekpyron You're probably running into https://github.com/vector-im/element-meta/issues/1339. Please give that some 👍 reaction love

Although it's possible you're talking about a different performance problem while all of those members were joining the room.

jakirkham commented 1 year ago

Congratulations on the merge! 🥳

Had a question about one aspect. AIUI there will still be two identities: one on Matrix and one on Gitter for each user (at least until PR https://github.com/matrix-org/matrix-spec-proposals/pull/2787 is complete). Is that right or am I misunderstanding something?

MadLittleMods commented 1 year ago

Had a question about one aspect. AIUI there will still be two identities: one on Matrix and one on Gitter for each user (at least until PR https://github.com/matrix-org/matrix-spec-proposals/pull/2787 is complete). Is that right or am I misunderstanding something?

@jakirkham Your understanding seems in the realm of correct (don't want to assume too much though).

@jakirkham-560313510fc9f982beb1a331:gitter.im exists on Matrix on the gitter.im homeserver. That will exist forever.

Currently, it's not possible to rename or merge that account with your other Matrix account (I assume you have some other Matrix account on matrix.org).

https://github.com/matrix-org/matrix-spec-proposals/pull/2787 is one possibility that could help this scenario though 👍

jakirkham commented 1 year ago

Thanks Eric! 🙏

Yeah that's right. Have @jakirkham-560313510fc9f982beb1a331:gitter.im and @jakirkham:matrix.org as well

Gotcha ok. Are there other proposals to be aware of as well?

MadLittleMods commented 1 year ago

Are there other proposals to be aware of as well?

@jakirkham I think that will hold the most references (anything linked from that).

A half-measure for example could be to mark an account as tombstoned in favor of another account. This way when someone tries to DM or mention you, their client would instead point them to them to the account you prefer. I am not aware of a MSC for this (feel free to create one)

jakirkham commented 1 year ago

Thanks for the pointers, Eric! 🙏

Another question: Am noticing that some Gitter DM's keep the history when ported over to Matrix and others don't (starting new blank DM's). Is there something I need to do on my end to make sure the history is pulled over from Gitter into the new Matrix DM's?

MadLittleMods commented 1 year ago

This section in the blog post might get you up to speed, https://blog.gitter.im/2023/02/13/gitter-has-fully-migrated-to-matrix/#where-is-my-message-history

If that doesn't answer it, come chat on https://matrix.to/#/#gitter_gitter:gitter.im. This issue isn't meant for ongoing support.

jakirkham commented 1 year ago

Ah sorry will do to that. Thanks for your help! 🙏

hiiamboris commented 1 year ago

List https://github.com/vector-im/element-web/issues/437 please. Existing clients are retarded enough as they are, but not supporting such simple feature is just unacceptable for modern software.

brianjmurrell commented 1 year ago

 Shutdown the IRC bridge, https://gitlab.com/gitterHQ/gitter-infrastructure/-/issues/262

This appears to be a dead link.

That said, what is the status of accessing Gitter from IRC now? No longer possible? Are we all going to have to have yet another stand-alone IM app/Web-chat page open now, on top of the forty-eleven that we already have?

This is a huge step backward if so.

hiiamboris commented 1 year ago

Excuse my ignorance as well please, but why is the request to implement one of the most convenient features of Gitter - ability to embed illustrations into text via ![alt-text](image-url) markdown code - not supported in any matrix client - is marked off-topic in a page titled "Gitter feature parity"?

leonardehrenfried commented 1 year ago

 Shutdown the IRC bridge, https://gitlab.com/gitterHQ/gitter-infrastructure/-/issues/262

This appears to be a dead link.

That said, what is the status of accessing Gitter from IRC now? No longer possible? Are we all going to have to have yet another stand-alone IM app/Web-chat page open now, on top of the forty-eleven that we already have?

This is a huge step backward if so.

I'm not sure if there is public instance running at the moment, but at least you can run your own matrix <-> irc bridge: https://matrix.org/bridges/#irc

brianjmurrell commented 1 year ago

I'm not sure if there is public instance running at the moment, but at least you can run your own matrix <-> irc bridge: https://matrix.org/bridges/#irc

That doesn't seem very scalable. Everyone wanting to use the IRC bridge that used to exist before this migration really should not have to stand up their own bridge.

Why was this feature simply forgotten about or ignored, particularly when it is straightforward enough to stand up a bridge?

leonardehrenfried commented 1 year ago

I'm in no way involved but I guess the reason is lack of resources. As a free service Gitter/Matrix have to prioritize some features over others.

Matrix is very, very bridgeable and perhaps someone already runs one.

MadLittleMods commented 1 year ago

@brianjmurrell Most of these are links to internal issues that you won't be able to see.

The IRC bridge was essentially a consumer of the Gitter API which as the past couple blog posts explain are no longer accessible.

There won't be a built-in alternative but there are a few IRC bridges you could run, https://matrix.org/bridges/#irc. matrix-ircd is similar to irc.gitter.im but it's pretty alpha, sadly.

If you want your Gitter room bridged to IRC, then there is a IRC bridge integration built-in but that has to be setup per-room by the room admins.

MadLittleMods commented 1 year ago

@hiiamboris I think you want to track https://github.com/vector-im/element-web/issues/437 and https://github.com/vector-im/element-web/issues/4044 which looks like you might have already found.

hiiamboris commented 1 year ago

@MadLittleMods yeah found those. But they're 6-8 years old (obviously neglected :), so it's unlikely they'll get attention without being listed above :/

MadLittleMods commented 1 year ago

@hiiamboris https://github.com/vector-im/element-web/issues/4044 looks reasonable enough to me that I think someone could move it forward.

Come chat in https://matrix.to/#/#gitter_gitter:gitter.im if you have further questions

brianjmurrell commented 1 year ago

The IRC bridge was essentially a consumer of the Gitter API which as the past couple blog posts explain are no longer accessible.

Yes, that is understood. What I am saying is that part of the migration to Matrix should have included rather than simply disregarded IRC bridge users. There are bridges available. I would submit that the migration should have included standing up a (single, central, public) bridge for all existing bridge users to migrate to.

MadLittleMods commented 1 year ago

@brianjmurrell While I agree that the Gitter IRC bridge was great and convenient, it's not core to Gitter (it's core to you). Bridging and interacting from IRC is still possible but you will have to take up the mantle.

almereyda commented 1 year ago

I see that

is striked out. I'm seeing runtime issues from the migration, where the account is unable to create or join certain rooms, which belong to GitHub organisations from it. Is there a feedback issue, some Gitter community (Discourse) or a personal channel where this could be discussed?

melvincarvalho commented 1 year ago

There are a lot of project histories stored in via bookmarked messages with the ?at query string.

Could they somehow be stored on gitter so that in future they can be easily spidered, and reused by automated agents? This has allowed things to get spidered, searchable, bookmarkable. Right now they seem broken links.

It important because for many projects this can act as a knowledge base, act as an automated FAQ, record contributes etc.

Perhaps gitter could make the history as it was before, but read only. With new messages going to matrix?

MadLittleMods commented 1 year ago

I'm seeing runtime issues from the migration, where the account is unable to create or join certain rooms, which belong to GitHub organisations from it. Is there a feedback issue, some Gitter community (Discourse) or a personal channel where this could be discussed?

@almereyda See the "What's changing?" section of the blog post. Best to discuss in https://matrix.to/#/#gitter_gitter:gitter.im

MadLittleMods commented 1 year ago

Could they somehow be stored on gitter so that in future they can be easily spidered, and reused by automated agents? This has allowed things to get spidered, searchable, bookmarkable. Right now they seem broken links.

@melvincarvalho There won't be redirects from the old Gitter chat message ID to the new Matrix event ID. But you can have the same setup as before by linking to a specific message (Share -> Copy the matrix.to link). This experience will also be better when the Matrix Public Archive comes to fruition:

[...] “Public static archive (indexed by search engines)” solution. We’re addressing this with the Matrix Public Archive but it isn’t quite ready yet to have a canonical instance for everyone to use yet. This piece will be an eager follow-up to today’s migration though.

-- https://blog.gitter.im/2023/02/13/gitter-has-fully-migrated-to-matrix/

GalaxySnail commented 1 year ago

There won't be redirects from the old Gitter chat message ID to the new Matrix event ID.

If I understand correctly, does it means that all old Gitter message links such as https://gitter.im/$namespace/$room?at=$message_id are now dead links and inaccessible forever?

MadLittleMods commented 1 year ago

If I understand correctly, does it means that all old Gitter message links such as https://gitter.im/$namespace/$room?at=$message_id are now dead links and inaccessible forever?

Initially while responding, I thought this was the case since doing it strictly would require a database and lookup service available forever which isn't ideal. But thinking more what's possible, I think we could extract the timestamp from the Gitter message ID and use jump to date in Element to get you to the right place.

This is now tracked by https://gitlab.com/gitterHQ/gitter.im/-/issues/9

waldyrious commented 1 year ago

That's great to hear :) and perhaps veering a bit off topic, the fact that this is now possible sounds like a great point in favor of transparent identifiers (for rooms, people, etc.) rather than —or in addition to— hash-like IDs.

calculuschild commented 1 year ago

GitLab/GitHub activity as a first-class citizen in a room’s side-panel

Any update on this? That should be the absolute top priority feature to restore, since it was the main distinguishing reason to use Gitter in the first place. I'm surprised there seems to be no motion at all there.