🔴 [Epic]: build blindnet devkit

[nweldev]

@milstan: I changed this issue's description so that tasks match architecture components. 2022-06-08 @milstan: I changed this issue's description to refine scope and set priorities. 2022-06-25

Description

According to Filip's comment, which is now considered normative, what we are building is called blindnet devkit. It replaces previous tools such as blindnet API & SDK, and Privateform which will continue to exist as one or more components of blindnet devkit.

This issue is a placeholder for high-level action plan for getting to build blindnet devkit. This issue is the TOP PRIORITY for Product work in the milestone Q3 2022.

Context

follows https://github.com/blindnet-io/devrel-planning/issues/24#issuecomment-1078928405

Intended outcome

blindnet devkit is a tool set for privacy-enabled connectedness. The key concepts of blindnet devkit are defined in normative documents:

• High Level Conceptualisation
• High Level Architecture

These documents represent a shared conceptualisation, that is (and should be) evolving, but it is considered normative pending any change.

The resulting blindnet devkit is a realisation of those normative documents as much as possible. It MAY be incomplete with regards to normative documents, but it MUST NOT be developed in contradiction with anything written in them.

Key Principles

• Seek to maximise: simplicity, interoperability, atomicity - ability to use only one part of the solution without the others i.e. no overhead). See comment.
• Focus first on the functionalities that are both (1) needed by the maximum number of prospective clients and (2) differntiating with regards to competition
• Design our system for interoperability with existing Open Source projects. Rather than reimplementing what they do, become their extension. The optimal design of our system is the design in which we extend the functionality of a group of Open Source projects that has the maximal existing adoption
• Reuse other Open Source projects whenever possible (unless there are explicit viable reasons in favour of not doing so)

The following steps will be better defined with @blindnet-io/engineering based on the conclusions of blindnet-io/devrel-management#27

1. create new generic Open-Source WebDev tools on top of our SDK (especially, allow a more declarative approach by creating a “technical” component library), and rethink the whole architecture of privateform frontend using them
   • Focus on building the generic WebDev tools, and then, rebuild privateform as a demo (using any WebDev library or framework, e.g. we can keep Angular if phase 2 is good enough)
   • Or focus on rebuilding privateform with lightweight tools (like lit, or even in VanillaJS), and extract generic WebDev tools from it progressively
2. build overlayers for the “vanilla-web-sdk” (i.e. results of phase 3) for all major frameworks (Angular, React, VueJS, etc.) + build / DevX tools (e.g. CLI, generators) if we see fit

Key Differentiators

The work on this issue seeks to develop distinctive differentiation from competition (as studied here). The key differentiators pursued are:

• Automation - ability to automaticaly process Privacy Requests
• Interoperability - ability of Systems to work together, using blindnet devkit, in the context of data exchnage

Features and functionalities that contribute to those key differentiators are considered prioritary.

Steps

Specs / Architecture preparation

• [x] https://github.com/blindnet-io/product-management/pull/596
• [x] #742

Development (< v0.8)

follow the MoSCoW method

Must

• [x] blindnet-io/product-management#710
• [x] blindnet-io/product-management#709
• [x] #762
• [x] blindnet-io/product-management#720

Should

• [x] #783
• [x] #781
• [x] #756

Could

• [x] blindnet-io/product-management#705

Further materials and actions

• [x] blindnet-io/communication-management#87
• [x] #782
• [x] blindnet-io/communication-management#111

Stakeholders

• @filipblnt - leader
• @noelmace - contributor & 0th customer
• @blindnet-io/engineering
• @milstan - product guidance

[nweldev]

@filipblnt do you have any related issue in product-management that should be linked to this one?

[milstan]

Just to note that the development of this new version (and in general any new code) should be done in alignement with the High Level Conceptualisation.

I will:

• [x] share more understanding about the high-level requirements and create a place-holder for discussing high-level architecture choices
• [x] organise a meeting to make sure everybody has good understanding of the High Level Conceptualisation, and answer questions

We also need ASAP:

• [ ] write a roadmap for this v2

[nweldev]

@filipblnt and I discussed this today.

• @m4rk055 and @filipblnt will discuss the basis next week
• @filipblnt and I will plan a meeting the week after that to better discuss DevX and web frontend architecture as I'll start working on blindnet-io/devrel-management#27
• we'll start writing an architecture plan and roadmap together (i.e. @blindnet-io/engineering & I) after @m4rk055 gets back from vacation (i.e. the week between 23 and 27 of May), and present it to everyone directly

[milstan]

Suggestion for Steps to be added:

1. Out of the total scope defined by High Level Comceptualisation, identify key areas to focus on in the short term. Now is the time to do this, when we transform conceptual documents into decisions and plans. IMHO, this means:

   • [ ] blindnet-io/communication-management#87
   • [x] Decide on key diferentiators to pursue (IMHO: simplicity, interoperability, atomicity - ability to use only one part of the solution without the others i.e. no overhead). See comment.

2. Design for complementarity with existing Open Source projects, meaning:

   • [ ] Identify popular open source projects that cover certain functional areas defined in High Level Comceptualisation
   • [ ] Design our system for interoperability with existing Open Source projects. Rather than reimplementing what they do, become their extension. The optimal design of our system is the design in which we extend the functionality of a group of Open Source projects that has the maximal existing adoption.

3. Design a roadmap for v2. The goal of this roadmap is to set priorities to favor differentiating functionalities (not covered well by competitors), in differentiating ways (simplicity, interoperability, atomicity), in maximal connection to maximally adopted existing open-source projects. Prerequisite:

   • [ ] blindnet-io/communication-management#111

The whole rebuild fits into the following goal (multi-objective optimization problem): generate maximal (contribution and) adoption in minimal time with minimal effort.

[milstan]

I've updated the description replacing TBD with concrete next step task. Many are already ongoing. This issue is now the top priority for product.

@filipblnt please:

• see if this issue should be moved to product-management or product-ideas, and if so act
• make sure the API/SDK remake is compatible with what we need here, or if not adapt
• let's coordinate and make sure the tech team is aware of the ongoing work (schemas, specs, architecture) on v2 and can align with it and get involved quickly
• add here the terminology clarifications (privateform/v2/blindnet devkit) we were talking about, so there is no more confusion

[milstan]

My understanding of the product positionning. The scope is related to privacy as a component of connectedness between humans and machines, humans and humans. The regulation is just a part of the need.

[filipblnt]

Regarding the product, here is a description of what it actually is today and what it will be tomorrow.

Today (23.05.2022)

We have two products:

• SDKs, intended for developers to use in their software to encrypt data end-to-end.
• Privateform, a client-server application for people and companies to privately collect data from their customers and manage data rights. Originally intended as a demonstration of the SKDs, in the meantime grown to contain more functionalities implemented in Privateform itself.

Tomorrow

We have one product: blindnet devkit, which is based on the HLC and HLA. It is a collection of SDKs, Restful APIs, and web components and interfaces intended for developers to implement data captures, their lifecycle management, and data rights management, in their own software.

When we say v2, we mean blindnet devkit. In v2, Privateform does not exist in the way it exist today; it can be built or integrated in external products/web sites using blindnet devkit, but that's all there is to it.

When blindnet devkit becomes available, existing clients will be reintegrated into it.

[milstan]

With regards to priorities, I've been thinking in the following way: Some systems already comply with some of the GDPR rquests by the simple design of their systems. In this regard, there are different levels of usefulness:

• Like If I am an e-commerce website, and I collect postal adresses of my users, then I have an interface for them to change adress. I don't need the devkit for it. On the other hand, if the user does not have an account, I don't have a service to let them formulate a request to know if I have data on them. Also for my own users, I don't have a feature allowing them to revoke consent, nor a feature to opose themselfs to particular kinds of tratment. So I want to continue using my existing interface, and use only a subset of the scope of blindnet devkit for things I don't have yet.
• I treat data of people who don't have an account with me at all - I have no direct relationship with them. Currently I only have a dpo e-maila dress where they can write to me, and then I'll manually treat the request. I need the whole blindet devkit to automate the processing of requests.
• I'm a real-estate agency. I treat data of my clients, but I have no information system with login (just a plain old website) - I just do it using an excell file. Yet I want to have an esy interface to collect Privacy Requests (and if needed transfer them to someone else to whom I've given the data as well), and I need an interface to see the request, reply to them and keep trace of request I've treated.

With the privateform v1, we've aimed at the clients who nad nothing and needed everything.

With v2/devkit, my intituion is we should start with functionalities that everybody needs (the largest possible need).

My suggestions is to think in terms of usfulness and prioritise the roadmap in that way.

[milstan]

Hey @noelmace , @jboileau99 , @m4rk055 , @TheKinrar

We spoke about adding to each milestone what we expect to acheive in it. Here is my proposal for the next few (covering month of August). It's just a proposal, feelf ree to change/challenge/refine as you move throught the work.

Ideally, we reach a meaningful complete realease at each milestone, and ideally by end of Auguest we have as much as possible full system. Then in September we cas see, as a result of testing what are we still missing, what we forgot, and what bugs we need to fix.

p.s. we also need to figure out how to test it - have some sort of demo.

DevKit v0.3 – 29 July

PRCI – PCE – DCI connected and can deliver, together, a non-mocked response to a transparency request. The user can make various TRANSAPRENCY.* requests and get different answers.

PCE can take in the configuration (RoPA and similar).

First steps of Data Access Component.

DevKit v0.4 – 12 August

PRCI covers all demand types. Data Subject can authenticate. DCI allows to view history of PRIV events, See recommended Responses, modify and send them. See past responses. Settings allow to set automatic responses to certain demands.

PCE and Data Access Component communicate. It is possible to fetch data for ACCESS demands.

PCE automates ACCESS, TRANSPARENCY (for logged and non-logged users), and REVOKE CONSENT. All those work in PRCI-PCE-DCI workflow.

DevKit v0.5 – 26 August

PRCI/DCI also send e-mail to Data Subjects when there is a response, so they come and see it (e-mail templates configurable at least from the code). All is multilingual.

PCE can take any PRIV event (except Data Captures) and deliver recommendations to any Privacy Request type.

Data Access / Storage can take instructions about data to modify/delete after responses to Privacy Requests or after loss of Legal Base.

DevKit v0.6 – 02 September

Large part of feature-set covered. All tested. Bugs, missing features identified.

[nweldev]

Testing and Demoing

Testing and demoing the frontend components is initiated in https://github.com/blindnet-io/privacy-components-web/issues/22, using Storybook and a new "CDN build". In v0.2, 3rd-party devs can already test the PRCI web component using Storybook on https://blindnet-io.github.io/privacy-components-web/, but there is still some more documentation needing our attention.

Testing and demoing backend components directly can be done with either Stoplight or Swagger, depending on what @m4rk055 opts for. I'd recommend using Stoplight for everything (modelling the API for mocking ahead of time, then replacing the OpenAPI definition with the one generated from the component's actual code afterwards, and linking this doc to the API root end-point) to avoid duplication and using too many different tools.

For the Data Access / Storage components, @TheKinrar will develop a demo implementation while creating a first version of the library, as this part is mostly about creating tools to improve the DevX when connecting the DevKit to an existing storage service, to avoid duplication of mappings and other storage related well-known matters.

I'll write content (including tutorials) to showcase these as part of https://github.com/blindnet-io/blindnet.dev/issues/26, and promote them as we go.

[milstan]

Thanx. This is unseful. It might interest @Vuk-BN .

[milstan]

👏 Thanks to all of your efforts, @Clementinev , @TheKinrar, @noelmace , @jboileau99, @m4rk055 this issue has been done.

The updated product-roadmap now reflects where we are at Q3, and hints future features our clients may expect. I'll be making a dedicated TOP PRIORITY issue for Q4 product development, that will be a summary of what we discussed.