search

status-im / status-web

Packages for building web applications in the Status ecosystem.
https://status-components.vercel.app
Mozilla Public License 2.0
81 stars 35 forks source link

Link previews #327

Closed felicio closed 1 year ago

felicio commented 1 year ago

What

How

Current:

Branch:


Relates:


felicio commented 1 year ago

My notes

felicio commented 1 year ago

Community

COMMUNITY_DESCRIPTION
Banner Yes
Photo Yes
Name Yes
Description Yes
Members count Yes (but computed)
Active users No
App link No (but computed)
Tags Yes (but computed)
Color Yes

Channel

COMMUNITY_DESCRIPTION CHAT_IDENTITY
Banner No
Photo No
Emoji Yes
Name Yes
Description Yes
App link No (but computed)
Color Yes
Community photo Yes
Community name Yes

User

CONTACT_CODE_ADVERTISEMENT
Banner No
Photo Yes
Name Yes
Description Yes
Emoji hash No (but computed)
Identicon No (but computed)
Social links Yes
App link No (but computed)
Color No

Group

MEMBERSHIP_UPDATE_MESSAGE
Banner No
Photo
Emoji
Name
Members count
App link No (but computed)
Color
felicio commented 1 year ago

@endulab @cammellos could you please confirm that we're currently not publicly broadcasting group chat details (i.e. photo, emoji, name, members count, color), and if so help me create and estimate issues for related repos?

Cc @pedro-et (for sanity check on the required fields)

cammellos commented 1 year ago

@felicio @pedro-et @John-44 we had conversations about this already, and we haven't settled on a decision, private group chat as it is now should not leak any metadata. We agreed that this is something that we would be discussing for post mvp.

endulab commented 1 year ago

@felicio we will handle only 1. community, 2. channel and 3. user links.

John-44 commented 1 year ago

@felicio @pedro-et @John-44 we had conversations about this already, and we haven't settled on a decision, private group chat as it is now should not leak any metadata. We agreed that this is something that we would be discussing for post mvp.

Sharing links to ad-hoc group chats (with group chat name, image, etc...) is out of scope for MVP, but highly likely to be something we will do shortly after MVP (while preserving the current fully private group chat functionality of course - users would effectively have the choice of converting a private ad-hoc group chat into a publicly sharable ad-hoc group chat if they wanted to). The design for such post-MVP 'sharable ad-hoc group chat' functionality is tbd.

Post MVP we will also be introducing many more types of URL links (e.g. links to request money including a message, links to individual messages in communities or 'open' ad-hoc group chats, etc..). But we don't need to worry about this for now, because we have more than enough items that are in-scope for MVP that we need to direct our energies towards.

John-44 commented 1 year ago

@felicio @pedro-et @John-44 we had conversations about this already, and we haven't settled on a decision, private group chat as it is now should not leak any metadata. We agreed that this is something that we would be discussing for post mvp.

Sharing links to ad-hoc group chats (with group chat name, image, etc...) is out of scope for MVP, but highly likely to be something we will do shortly after MVP (while preserving the current fully private group chat functionality of course - users would effectively have the choice of converting a private ad-hoc group chat into a publicly sharable ad-hoc group chat if they wanted to). The design for such post-MVP 'sharable ad-hoc group chat' functionality is tbd.

Post MVP we will also be introducing many more types of URL links (e.g. links to request money including a message, links to individual messages in communities or 'open' ad-hoc group chats, etc..). But we don't need to worry about this for now, because we have more than enough items that are in-scope for MVP that we need to direct our energies towards.

felicio commented 1 year ago

@felicio we will handle only 1. community, 2. channel and 3. user links.

@endulab please,

endulab commented 1 year ago

@felicio

felicio commented 1 year ago

@endulab thank you, and to follow up

Can you elaborate ? Installation package already registers custom link "status-im".

Let me try to rephrase then. Will, or can, desktop open and handle links starting with https:// like mobile does? Thus rendering status-im:// scheme obsolete for cases like "Open in Status" (see https://github.com/status-im/status-desktop/issues/6612) and redirecting users straight to the app if installed, and landing on the website only if not.

which contains also community id

Which technically is the community's compressed public key later then used for content topic and symmetric key, correct? If not, could you make a reference to the code?

endulab commented 1 year ago

@endulab thank you, and to follow up

Can you elaborate ? Installation package already registers custom link "status-im".

Let me try to rephrase then. Will, or can, desktop open and handle links starting with https:// like mobile does? Thus rendering status-im:// scheme obsolete for cases like "Open in Status" (see status-im/status-desktop#6612) and redirecting users straight to the app if installed, and landing on the website only if not.

Oh, I see. The app can't do it right now, but we can try to do it this way. However, I understand that if we have universal links, the app will be opened directly w/o need of our website. Is it the workflow we want to have ? I will have to do some research how to do this. Also, we need this functionality on Windows. Let me know if we need this feature; I will create the ticket then.

which contains also community id

Which technically is the community's compressed public key later then used for content topic and symmetric key, correct? If not, could you make a reference to the code?

Yes, the community key in the community channel key is a 33-bytes compressed public key. However, in order to expose keys, we use a different compression, which is called "multiformat" compression - zQ3sh........ We use multiformat to expose userId and communityId. Eg. Status community link is: https://join.status.im/c/zQ3shf8EvhRKHSc3skgtupULuLdWh59S5czZkvS69P6rJ7aed To keep consistency we'll have to use multiformat in channel id: Instead of: "0x036f5a79a5..." have "zQ3shf8Ev" I hope that makes sense.

felicio commented 1 year ago

@John-44 @pedro-et could you please confirm that desktop should open and handle links starting with https:// like mobile does? Thus redirecting users straight to the app if installed, and landing on the website only if not.

Cc @endulab

John-44 commented 1 year ago

@John-44 @pedro-et could you please confirm that desktop should open and handle links starting with https:// like mobile does? Thus redirecting users straight to the app if installed, and landing on the website only if not.

Cc @endulab

Yes, both desktop and mobile should open and handle links starting with https:// , it's really important that they do this.

@endulab the URL format is going to change, @felicio @prichodko are working on some improvements to the URL format so I'll set up a call so we can discuss later in the week

felicio commented 1 year ago

@John-44 please, could you review the individual "Matrix: Community" and "Matrix: Channel" sheets at https://docs.google.com/spreadsheets/d/1JD4kp0aUm90piUZ7FgM_c2NGe2PdN8BFB11wmt5UZIY/edit?usp=sharing and let me know if data encoded and total url lengths, from the product point of view, are worth not using and exposing public keys to servers?

While the final encoding may undo some of the compression causing the total number to go up or down by ~5%, I do believe it can be stabilized and further reduced (e.g. LZ compression, text trimming) but still placing it somewhere between 150-250 characters.

Cc @prichodko @fryorcraken

felicio commented 1 year ago

felicio please check out https://github.com/waku-org/js-waku-examples/blob/master/examples/store-js/index.html Note that currently the API is such that more JS code is downloaded that what is striclty necessary, I want to improve the way the bundles are split so that minimum code needs to be downloaded by the user.

– Status, web, 0x6916cc7e7db8fe0ecbe2731a9ed9c7758a5dc15f4f1629e29fbe288eb94be1e0

@fryorcraken.eth | Waku Hi, I assume this has to do with today's call where I mentioned the use of store protocol within a lambda like env (i.e. server, node.js, relatively enough space) and where the exec should take only couple of seconds to fetch the preview details. My question was if you guys would consider it a valid use case for a waku node to support REST API. And if you would have any concerns regarding the load/scalability even with caching configured (e.g. at max 1 request per community/channel/user per let's say 15 mins., depending on users sharing and clicking the reference link). But thank you for the above info anyway. Cc LNSDev

– Status, web, 0xe52f8415d9b20c12aacda745763fcc42a8c02480869f05969c4d28b3c54c4c7c

felicio commented 1 year ago

Ok. let's wait @felicio.eth 's input because in Status Desktop he mentioned the usage of caching and NodeJS so I am a bit confused as the usage of js-waku would permit a frontend/client/browser only solution.

– https://discord.com/channels/864066763682218004/1067545488934588426/1068355176173424670

@fryorcraken.eth | Waku these links are related to

  1. previews (e.g. on social or other IMs displaying meta info and an image)
  2. website (for environments without native app and when circumventing deep linking)
  3. deep linking

and it is the previews that I mentioned for which some work on the server side is needed. And if they were to be based on any data from waku now (e.g. images), which hasn't been definitively ruled out yet but explored for the purposes of the feature analysis nevertheless, it'd require such set-up.

backend logic makes the solution quite centralized and Status-dependent Hosted, or self-hosted that I understand to be the case even with the current solution (https://github.com/status-im/universal-links-handler) served from our servers.

Are you aware of a decentralized way that can dynamically generate a html doc prior returning it to the client? Would you have an example for reference? I've heard about ipns (not ifps), but didn't spend time researching it.

data from a store query

The rest api comment I made, so the peer connection and disconnection wouldn't have to be awaited for such a short-lived op.

link points to a bridge

Could you please elaborate?

I believe that the resulting approach will depend on following factors:

  • url length with some data already encoded
  • practical use
  • speed
  • design
  • privacy
  • level of centralization and control
  • and whether any part puts us in the role of a content provider

As soon as I post the summary to aid with the decision in https://github.com/status-im/status-web/issues/327, I'll mention you as well.

Does this make things more clear?

– https://discord.com/channels/864066763682218004/1067545488934588426/1068477948023296032

link points to a bridge Could you please elaborate?

felicio I am talking about hosting JS code on IPFS and have the invite link pointing to this code, possibly via a IPFS bridge. However, I guess this is a level of decentralization that is not necessary for Status MVP.

Does this make things more clear?

Yes thanks, especially regarding social preview.

As soon as I post the summary to aid with the decision in https://github.com/status-im/status-web/issues/327, I'll mention you as well.

Sounds good.

– https://discord.com/channels/864066763682218004/1067545488934588426/1069444784042807407

felicio commented 1 year ago

@Samyoul gave me a great feedback:

@Samyoul , @cammellos , @richard-ramos , @iurimatias , @saledjenic , @jakubgs , @fryorcraken, @prichodko I was going to propose an extension to https://github.com/status-im/specs/pull/159 in form of a single url search parameter carrying the serialized, compressed and encoded content as per "Matrix:*" sheets at https://docs.google.com/spreadsheets/d/1JD4kp0aUm90piUZ7FgM_c2NGe2PdN8BFB11wmt5UZIY/edit?usp=sharing. However after considering the above, I now propose

For the url scheme:

For clients:

And for the infrastructure:

Please, would you agree?

NOTE: What I'm proposing already has some POCs, so if you'd like to validate something let me know.

jakubgs commented 1 year ago

If we are re-doing deep links the we need to revisit using hash arguments to preserve privacy.

Using paths or URL arguments means the paths end up on the backend side. If we use hash arguments we prevent whatever is hosting the deep-linking UI from knowing what people are actually linking.

jakubgs commented 1 year ago

As for the /cc/[compressed_community_public_key]/[channel_uuid] format. As long as we can generate it on the client side in the browser and use hash arguments that's fine.

felicio commented 1 year ago

@jakubgs thank you,

revisit using hash arguments to preserve privacy ... If we use hash arguments [/c/#[compressed_community_public_key]] we prevent whatever is hosting the deep-linking UI from knowing what people are actually linking.

that is true, even for whatever backend/server is generating the previews, however without an access to the public key I cannot fetch content like profile images from Waku nor verify the encoded content in url. Because if it contained a signature the public key could be recovered anyway.

On the other hand, and considering "IPNS's hashes of public keys instead, probably of those used for chatting" I've mentioned above, I think the verification could be done by comparing the hashes after visiting the urls (and in our apps even before clicking or resolving).

But previews outside our apps would still need to be considered untrusted, I believe.

@rymnc @s1fr0 it might be a reach mentioning you on this issue, but I'm convinced your take on the matter:

would be helpful.

rymnc commented 1 year ago

@felicio, thanks for the tag - let me get up to speed with this issue and let you know soon

felicio commented 1 year ago

Following has been decided to address above mentioned concerns in short term,

limiting proposals to following two,

@John-44 please, could you confirm with 👍 or otherwise comment below? Also, do let me know if I should schedule a call and clarify anything.

Cc @prichodko

Note: Mind that some social platforms or IMs vissualy trim the urls, so urls would actually not be displayed in full length (see "Limits:*" sheets at https://docs.google.com/spreadsheets/d/1JD4kp0aUm90piUZ7FgM_c2NGe2PdN8BFB11wmt5UZIY/edit?usp=sharing)

Note: Without sharing public keys with hosting servers some data like profile photos or banners cannot be included in the previews (see "Matrix:*" sheets at https://docs.google.com/spreadsheets/d/1JD4kp0aUm90piUZ7FgM_c2NGe2PdN8BFB11wmt5UZIY/edit?usp=sharing)

Note: encoding small data has actually an opposite effect due to serialization and compression headers, which could however be optimized to shave off couple of characters but left as-is for simplicity

felicio commented 1 year ago

https://github.com/status-im/status-web/issues/327#issuecomment-1429283263

Per request,

Cc @John-44

felicio commented 1 year ago

At the beginning I pointed out,

Please, where are these being addressed/tracked now?

Cc @John-44

endulab commented 1 year ago

@felicio I created 2 tasks for MacOS and Windows: https://github.com/status-im/status-desktop/issues/9549 https://github.com/status-im/status-desktop/issues/7957 I will catch you later to discuss details.

felicio commented 1 year ago

In addition to https://github.com/status-im/status-web/issues/327#issuecomment-1429996678,

Cc @John-44

John-44 commented 1 year ago

@endulab @cammellos probably obvious (so apologies for stating the obvious) , and I'm not sure if this is the correct place to mention this, but the generation of this URL needs to happen in status-go so the same URL generation code can be used by both Status Desktop and Status Mobile

felicio commented 1 year ago

@felicio I created 2 tasks for MacOS and Windows: status-im/status-desktop#9549 status-im/status-desktop#7957 I will catch you later to discuss details.

~@iurimatias~ @cammellos, please could you create the tasks too but for our supported mobile versions and share the links here afterwards? See https://github.com/status-im/specs/pull/159 for changes to the URL scheme.

Cc @pedro-et

cammellos commented 1 year ago

@felicio apologies for the delay, created issue on mobile side https://github.com/status-im/status-mobile/issues/15325

jakubgs commented 1 year ago

I see we have introduced a <signature> that is behind a # hash argument. Not sure what's the purpose of the signature, but that still means we would still be receiving things like [compressed_channel_key] or [compressed_user_key or ens_name] via the URL on the backend/storage side of the preview page. Are we sure we want to do it like that?

The goal is preserve annonimity of users, and this scheme does allow for correlating the IP with most of the URL on backend.

felicio commented 1 year ago

@jakubgs

\<signature> that is behind a # hash argument

That, as well as public keys or ENS names.

what's the purpose of the signature

To ensure integrity of the encoded content. More on that at https://github.com/status-im/specs/pull/169#discussion_r1115990922.

we would still be receiving things like [compressed_channel_key] or [compressed_user_key or ens_name] via the URL on the backend/storage side of the preview page.

No, as per above. Also, since we're using secp256k1, public keys can be recovered on the client side from the encoded content and the signature.

correlating the IP with most of the URL

  1. Was your concern only public keys? Which has been hopefully addressed above.
  2. What other content before the hash do you consider not preserving of anonymity?
  3. Who did you discuss this with? And what agreement have you reached?

The now encoded content could also be moved after the hash. Which would however mean that none of it could be used in social previews/cards.

Cc @John-44

jakubgs commented 1 year ago

Was your concern only public keys? Which has been hopefully addressed above. What other content before the hash do you consider not preserving of anonymity?

Any data that identifies what channel/person/group the link is for.

Who did you discuss this with? And what agreement have you reached?

I discussed it with myself. My understanding is that the things before the hash still reveal information about what channel/person/group is being linked and that can be correlated with IP and also client data.

Is my understanding correct then that this decision is between preserving privacy and not being able to do previews, and not preserving privacy but having fancy previews?

felicio commented 1 year ago

@jakubgs

as we have discussed several times.

– https://github.com/status-im/specs/pull/159#pullrequestreview-1341322442

Based on the above, I thought you have discussed this with others thus could provide more context as to who and why exactly.

If not, no worries.

My other questions were just to figure out whether your concern was only public keys or other (encoded) content too, and simply note the implication.

@John-44 please, could you leave a comment here addressing the following concern @jakubgs has raised? I could then make changes accordingly, if necessary.

correlating the IP with most of the URL ([not public keys but display name or description, for example]) on backend

– https://github.com/status-im/status-web/issues/327#issuecomment-1466003615

John-44 commented 1 year ago

Yes there is a trade between privacy / adoption here, because yes a sufficiently determined attacker could perform a correlation attack (however to do so in this instance would not be trivial). This has already been discussed a bunch, and it was from this discussion that the decision to not have images in unrolls (so the keys would be shielded by the # from the server) but to have URL specific metadata in unrolls (at the cost of exposing this metadata to the server) was made.

It might be a good idea to log a feature request in https://status-desktop.featureupvote.com/ for post MVP that is titled something like "give users option to share URLs without metadata for privacy purposes". Then we can give the user the choice of whether or not they want to share A) URLs that unroll nicely with meta info but where the server also sees this meta info or B) URLs that don't contain the meta info but which also don't contain any URL specific content in the unroll.

Such as feature would not be for MVP because we are busy trying to cut scope, and we really can't afford to add scope at the moment. For MVP the decision has been made to go with the trade I've described above

jakubgs commented 1 year ago

Very well, if this is a conscious trade off that people are ware of, fair enough.

jakubgs commented 1 year ago

I think it's fine if we add an option to have the privacy preserving URLs without nice previews for users to pick in the UI later on, as long as we actually remember to do that. And not just keep it as a "nice to have" forever.

felicio commented 1 year ago

Closing in favor of https://github.com/orgs/status-im/projects/91 for tracking.