Tribler / tribler

Privacy enhanced BitTorrent client with P2P content discovery
https://www.tribler.org
GNU General Public License v3.0
4.86k stars 451 forks source link

Digital Basic Identity - EU EBSI gateway and "regie op gegevens" #6023

Closed synctext closed 3 months ago

synctext commented 3 years ago

Topic is determined: Digital Basic identity, see https://www.tweedekamer.nl/kamerstukken/brieven_regering/detail?id=2021Z02985&did=2021D06488

Estimated timeline (tentative)

First sprint target of 2 weeks: see above

sjacobino commented 3 years ago

@synctext you mentioned the groundwork laid by Rowdy for digital identity. Is there something I can start looking into?

sjacobino commented 3 years ago

SSI Pull request

https://github.com/Tribler/trustchain-superapp/pull/60/files

synctext commented 3 years ago

2 week pause due to circumstances. Code compiles locally, EBSI reading first pass done, pull request Rowdy review on ToDo list. Recent find: Self-Sovereign Identity (SSI) en Attribute Based Credentials (ABC)

synctext commented 3 years ago

For more context, in 2017 we talked about the "decentralised economy". Identity is the foundation. Others are now also doing this in a token hype manner: https://fetch.ai/wp-content/uploads/2019/10/Fetch.AI-Economics-white-paper.pdf This team actually deployed an IPv8 clone: http://www.ifaamas.org/Proceedings/aamas2021/pdfs/p1037.pdf More work to read, solid data pods from MIT and web founder.

The current generation of thesis students in the lab. Student Thesis focus
@InvictusRMC Working SSI. operational attestations. Revocation problem and QR codes. #5576
@jwbambacht Reliable enough for money. Effortless EuroToken transfer in chats. UDP packets always arrive. #6029
@awrgold production-ready DAO. Shared ownership of BTC. Democratic decision making process. #5313

Promising thesis direction: Create an architecture in which Big Tech alternatives can emerge, market failure can be corrected, anti-competitive practices are addressed and consumers win their privacy back. Protecting privacy of users in the age of AI and digital IDs.

ToDo sprint: compiling new Superapp code, revocation of Rowdy code, read litarature. Future sprint: first draft of problem description, EBSI enrolment

sjacobino commented 3 years ago

Start on the problem description: Self-Sovereign Shared Identity

Use case is SSI for group subscriptions which includes data sharing and some key management.

SSSI Problem description.pdf

Inspiration for another tangent: fair compensation scheme for personal data in personal vault https://cordis.europa.eu/event/id/148435-bdve-webinar-bringing-back-personal-data-value-to-the-rightful-owners https://cordis.europa.eu/project/id/871755

synctext commented 3 years ago

Good readable first version of problem description. Just replace "family plan" with "legal entity", then it becomes economic relevant and key for SSI. Authorisation within huge companies or hierarchical government is unexplored. Still good to keep in mind we can go in any direction, like ask Schiphol for real on-site tests. Another direction is deep alignment with @jwbambacht his work. Your thesis would focus entirely on getting an encrypted address book (e.g. social graph, friend list) into the Superapp SSI part. Scientifically this is a validated public key infrastructure that would be actually operational and fully decentralised. Signal messaging app has central servers, no such DPKI system exists. btw about selling your own data. The idea of mixing capitalism of selling data with social value such as data sovereignty is old. The first to publish this, conclude that its a bad idea was these smart people: https://blog.oceanprotocol.com/on-selling-data-in-ocean-market-9afcfa1e6e43 Conclusion, nobody wants that data. This model can't compete against Big Tech who owns you (Mark Zuckerberg called early Facebook users dumb f***s for sharing data when he was just 19 years old).

ToDo idea for coming 2 week, align at the source code branch level with Joost and Rowdy. Get an rvig.nl email; get account for EU EBSI; get a working cmdline Python script working or directly integrate with Superapp.

synctext commented 3 years ago

Possible sprint idea for coming 2+ weeks. Goal: get operational coding experience. Make first Pull Request on the Superapp. For instance, add address book sharing in PeerChat. Any contact entry you have locally (nickname+public key) becomes available to share with your friends. Status: emulator works, next step is old S4 hardware attempt. Search for good thesis focus.

Possible goal: - Future sprint: EBSI integration, https://api.ebsi.xyz/docs/ - Live test, beyond Beta code; international .DE? - Expand upon @jwbambacht of "waardeoverdracht" into a _Secure Data Sharing_ primitive. With personal data vaults + exchange.
sjacobino commented 3 years ago

Preview contact sharing

Uploading IMG_4267_MED_1_1.mp4…

synctext commented 3 years ago
sjacobino commented 3 years ago

Preview contact sharing in gif form:

ezgif com-optimize

sjacobino commented 3 years ago

To do in contact sharing feature:

sjacobino commented 3 years ago
sjacobino commented 3 years ago

Share contacts PR: https://github.com/Tribler/trustchain-superapp/pull/77/commits

Small feature that allows you to send one of your contacts as an attachment in a conversation. Recipient can click on the contact and add it to his/her list.

sjacobino commented 3 years ago

Next steps:

Current vision/goal

synctext commented 3 years ago
sjacobino commented 3 years ago

Because of electricity/internet problems, no video call on 23/07/2021.

Updates:

sjacobino commented 3 years ago

Report from last month:

In the process of connecting the SuperApp with EBSI.

V1 api is implemented in the code but EBSI has already moved on to V2. V2 has turned several REST APIs into libraries to be used locally. Mobile libraries are not yet available, so I have implemented a CLI in TypeScript with the available EBSI api with some key functionalities (including getting access tokens to write) to the public test environment. Working tree: https://github.com/sjacobino/trustchain-superapp/tree/ebsi

Roadmap for translation between IG-SSI and EBSI is in the works including working on the translation code. See notes: https://drive.google.com/drive/folders/1OBEOV7G1Pb1ZbakTTg6Y42Vu1K_uX-ha?usp=sharing

Probably 1 more week to work on translating between IG-SSI attestations/presentation to EBSI Verifiable Credential/Presentation. But because mobile libraries are not available, it will not be possible to test in-app.

So likely to shelf this part for the time being and start on Data Vault.

sjacobino commented 3 years ago

EBSI docs, api and libraries currently not in sync. Conflicting information on the different platforms which is making it impossible to make a working connection.

E.g.

Screen Shot 2021-08-30 at 08 55 41

So let's fix that:

Screen Shot 2021-08-30 at 08 56 03

Discrepancy in information, in this case the parameters of a DidAuthResponseCall.

Sample code on NPM site: https://www.npmjs.com/package/@cef-ebsi/did-auth

Create Auth Response did-auth_0_0_12

Sample code on EBSI Doc: https://ec.europa.eu/cefdigital/wiki/display/EBSIDOC/DID+Authentication+Library

Screen Shot 2021-08-30 at 13 54 30

Function definition in DIDAuth.ts

Screen Shot 2021-08-30 at 14 10 45
Status EBSI v1 EBSI v2
Read access :heavy_check_mark: :heavy_check_mark:
Write access :heavy_check_mark: :x:
sjacobino commented 3 years ago

Personal Data Vaults Introduction.pdf

synctext commented 3 years ago

Personal Data Vault (brainstorm thesis direction):

sjacobino commented 3 years ago

Digital_Basic_Identity_MSc_Thesis___TU_Delft-3.pdf

sjacobino commented 3 years ago

Progress on bug report:

Screen Shot 2021-09-15 at 09 14 42

synctext commented 3 years ago

Storytelling: immediately reveal the plot and don't write it as a slow tutorial: Secure sharing of personal data is an unsolved problem. Switch now to thesis format, style, and thesis wording. Grand Problem Solving: we give control back to users and erode the power of Big Tech with their $7.6 trillion market cap. More professors like Bart Jacobs are now interested in fighting Big Tech through deployed alternative technology. Academics going beyond rhetoric! Like tobacco industry, social media is not going to fix itself: More than 30% of teen girls who 'felt bad about their bodies' said Instagram made it worse according to Fox News. This is a pretty angry motivation, but I believe the observed facts warrant this.

Is there any difference between an open wallet and the personal data vault? unlimited smartphones, please exclusively apply zero-cloud infrastructure! Support REST api?

sjacobino commented 3 years ago

Data Vault Demo Apk: https://drive.google.com/file/d/1qPcO9RoSYthnZXOy6q82dVZlihtIFeji/view?usp=sharing

sjacobino commented 3 years ago

What's going on in the demo apk:

Initial data vault directory on the device that is accessible by other devices via the DataVaultCommunity. Simple access control that takes any verifiable attestation.

The problem now is that messages including attestations don't seem to get to the other side. Solution is likely to copy peer chat's attestation chunk approach, and reassemble back at the other endpoint.

Defining access policy at file or directory level is viable. Files with undefined access policy inherits policy from parent directory.

Biggest challenge seems to be structuring the filesystem to reflect graph-like nature of the linked data.

synctext commented 3 years ago

please add this public key: 4c69624e61434c504b3a52904df841596dff8149fc41623b91b1149f92fe37f2c1c40c5ffb2867b7a75438e2694ab58b3c280578b80f348ca62fda10372581e3365cbd87c0551f3bc007

The problem now is that messages including attestations don't seem to get to the other side.

Progress: UDP packet drop is an issue. Large packets always drop? Scientific side: infrastructure of sharing data requires trustworthy data exchange primitive which is robust.

We have incoming UDP packets inside IPv8. Do we have a Diffie-Hellman key exchange, session establishment, signed UDP packets, and a 'connection'? Transmit & store encrypted? e-IDAS high compliance: :x: Personal information is always encrypted and protected See new 2021 update of eIDAS: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM%3A2021%3A281%3AFIN Note that "Electronic ledgers" is mentioned 23 times in official upcoming European Commission text. Pull request on "Joost code", no private icon.

sjacobino commented 3 years ago

Report on progress before leaving on the 13th Further updates via ticket.

synctext commented 3 years ago

example for thesis? Secure sharing of personal data is an unsolved problem. Store in your data vault, facilitate sharing and test revocation: diploma.

Data vault prototype for DNA data around eye melanoma (severe cancer), https://github.com/LedgerProject/GeneCoop/blob/a759c0368c1e2c3740a84c23db4ce1cb1e8a5db1/Demonstrator/Documentation/Technical_Design/demonstrator_tech_design.md

synctext commented 3 years ago

Fresh related EU work: https://europa.eu/europass/en/european-digital-credentials-learning-interoperability

sjacobino commented 2 years ago

No real productivity in last 2 months.

Some further improvements on the EBSI CLI using updated libraries. Problems are still difficult to iron out because error messages from the API are sometimes unclear.

What works:

Not managed to get working:

Screen Shot 2021-12-15 at 11 50 28

In the end we need the mobile (Android) libraries from EBSI to integrate it into the SuperApp, but those are not planned for EBSI v2 ( https://ec.europa.eu/cefdigital/wiki/display/EBSIDOC/Users+Onboarding+API#UsersOnboardingAPI-MobileWallet(NotYetforEBSIV2.0) , there is a roadmap somewhere that states it clearly but I cannot find it now)

Personal course adjustment necessary, turning focus to Data Vault using verifiable credentials for access control.

synctext commented 2 years ago

Last progress meeting was 2 months ago. Running code :clap:

EBSI v1 was everything server-side. EBSI v2 is typscript based. Progress of recent week is that EBSI team did some changes, updates, and repairs. More is operational now, JSON web tokens now appears functional. EU EBSI Mobile roadmap for mobile :zap: : EBSI compliant data model is not available Upcoming day: file a bug report with EBSI team on you latest error. Prior DidAuthResponseCall bug is now fixed :heavy_check_mark: Detail: EBSI W3C did model versus the Rowdy/Quinten verifiable credentials.

Master thesis content: The first SSI data vault deployment.
First description draft: Our architecture, called HardGlass, gives users back control. Everything that happens with your data becomes visible. Data read permission can be granted securely using encrypted communication to selected parties. Revocation is executed in real-time. At a fine-grained level the user sees all access to his/her data in real-time. Something, something, more is needed: scalability to 1 TByte, magnet&music usage, diploma works, friend certificates, distributed social, friend-of-friend access, etc. Upcoming sprint: improve prototype, think more about your beyond state-of-the-art.

ToDo: poster outline.

sjacobino commented 2 years ago

Brief description of my experience working with European Commission EBSI code:

Working with EBSI code has been a bit cumbersome at times. In some cases example code provided on the npm page is incomplete. If the code is copied and pasted some parameters might be missing. For instance, the sample code for creating SIOP authentication responses (https://www.npmjs.com/package/@cef-ebsi/siop-auth) does not show how to structure include claims/verifiable credentials/verifiable presentations. You have to combine information from the documentation page to get a complete picture. This is the case with most of the EBSI libraries available on npm.

Some information however is nowhere to be found. E.g how to construct a json-rpc request to insert a new DID document (https://ec.europa.eu/cefdigital/wiki/display/EBSIDOC/DID+Registry+API). The parameters used: identifier, didVersionInfo, timestampData, and didVersionMetadata are not explained anywhere leaving you to guess what they are.

As mentioned before, API error messages are often non descriptive and do not state very clear what the error is or what has to be changed.

synctext commented 2 years ago

perfect info! please post a reproducible error to EBSI and provide details. Then we can escalate them upwards.

sjacobino commented 2 years ago

EBSI+TrustVault poster draft to be discussed.

Main questions: Too abstract/not technical enough? More (specific) technical illustrations? Does it get the message across (especially if orally presented)?

EBSI+TrustVault Poster.pdf

synctext commented 2 years ago

:+1:

sjacobino commented 2 years ago

Trustvault + Essif final poster for meeting AFM

TrustVault+ESSIF Poster-4.pdf

sjacobino commented 2 years ago

DataVault prototyped advanced in the last 2 weeks:

The process now goes as follows

  1. You get a list of peers from which you can request files
  2. You send a request to see which files are accessible to you with the attestations that you have
  3. You get a filtered list back from your peer of files you can select from
  4. You choose a file and it is retrieved

The advancements are

  1. Smooth workflows
  2. Attestations are actually verified but no distinction is made yet between the different types of attestations
  3. Individual files can be made public so that no credential is required to get them. This menu is will be expanded to set granular, more complex access rules (instead of just public or not)
  4. To minimize overhead, attestation only need to be sent at first request. Subsequently a token (derived from the attestations) is used to retrieve subsequent files. (Some interesting caching needed here)
  5. Some interface improvements as well, with the goal in mind to migrate everything under one Value Transfer roof.

See Slack for apk file.

synctext commented 2 years ago

Very productive sprint !! :tada: Real productive software development inside superapp: 6 weeks. EBSI: 6 weeks. Key selling point? data sovereignty?? Current state-of-the-art is MIT solid. random test sample of this prototypes; horrific user on-boarding experience, but good luck trying to do this better :hourglass_flowing_sand: Connecting a PC browser to storage will never bootstrap (critically requires support from the monopoly browser). For Android its also hard to create automatic background cloud storage. You first need an identity and (free??) storage. Fragile... MIT Solid is server-based and has a high probability to re-centralise all data at new players (Big Tech 2.0).

sjacobino commented 2 years ago

This image shows the policy editor page, when you click on an image/file in your data vault. You see the different policies, in this case a read policy and a read+write policy with the rules defined for required credentials for access.

These two images show how you can edit a rule for credentials

To get to the core of the thesis: giving users full control to define how 3rd parties can retrieve files in his/her vault. This is done by defining what is basically a decision tree for required combination of credentials. This is done granularly at file and folder lever. Credentials can be self-issued by the user (akin to accepting a friend request) or by trusted 3rd parties (IG-SSI SuperApp credentials, and if the stars align, even from EBSI). The goal is to combine sovereignty over identity with sovereignty over data in general. No more giving your phone to someone to see some pictures and hoping they don't swipe left too many times. No more pictures on the servers of Facebook and Instagram but all on your own device.

synctext commented 2 years ago
sjacobino commented 2 years ago

Let's try to find out more about the wallet conformance testing https://ec.europa.eu/cefdigital/wiki/pages/viewpage.action?pageId=455606439

sjacobino commented 2 years ago

This sprint in the end was all about UI/UX taking a lot of time. The browser now looks like a proper file system and has file system functionalities. Files can now inherit access control rules from their parent folder. Having some issues still with loading remote files. So EBSI conformance testing is still on the to-do list.

https://user-images.githubusercontent.com/5730521/155093185-20054916-e12a-4e80-92a7-8b964f07e6fd.mov

synctext commented 2 years ago
synctext commented 2 years ago
sjacobino commented 2 years ago

EDIT: Managed to complete this one. Needed to include a one time token from https://app.preprod.ebsi.eu/users-onboarding/ in the request

Similar error messages as a while back. Not really clear what the issue is:

Screen Shot 2022-03-11 at 09 45 43
sjacobino commented 2 years ago

EBSI is using ECDSA with the secp256k1 curve which is being deprecated by most libraries. The trend is to switch to either 256r1 or to EdDSA. Not really future proof. See https://jwt.io/libraries

sjacobino commented 2 years ago

Last sprint:

-Authentication and so finally succeeded. Current stumbling block is interacting with smart contract end-points. Documentation and examples still very difficult to navigate:

Screenshot 2022-03-29 at 14 50 50

or

Screenshot 2022-03-29 at 14 52 13

Is it it hex or base64????

sjacobino commented 2 years ago

EBSI conformance plan:

https://ec.europa.eu/digital-building-blocks/wikis/display/EBSIDOC/EBSI+Wallet+Conformance+Testing

synctext commented 2 years ago

With 57 Million :euro: in funding for EBSI Large Scale Trails. The EU wallet is only growing in importance as critical infrastructure for next year :astonished: Terms for selling your work: "EBSI-compatible data vault" or "Secure EBSI data sharing". The latter indicates more the access authorisation and generic framework that you provide. Focus on https://EBSI.readthedocs.io/ and pioneering foundations for others? Your thesis sells "accelerating the EU ID movement by providing the first EBSI-certified wallet and secure data sharing. Our work is targeted to be the reference implementations for others to build upon or improve (open source).", not just another photo+data sharing app.

Currently hotels all are by law required to copy your paper-based passport. What will EBSI give them? It is like the infamous Corona scanner, it remembers how many identities are checked? Or is the verifiable credential proof stored permanently?

sjacobino commented 2 years ago

Again some headaches in the conformance testing process. Unable to do significant testing for over a week. It seems one issue is resolved and another is created.

Screenshot 2022-04-21 at 09 37 11 Screenshot 2022-04-21 at 09 37 26

EBSI conformance scenario 2 (Requesting verifiable attestations) of the Conformance program is almost complete. Again, without blocking issues on the EBSI side, scenario 3 (presenting attestations) should not be too difficult (1 week max).

Made some improvements on the data vault during the EBSI down time:

On the data vault, mostly usability/UI improvements needed. Performance improvements (higher transfer rate) would be nice but might be too big a task.

Shifting focus towards writing to have a rough draft in 2 weeks.