REFACTORING

[Jean-Baptiste-Lasselle]

private content in : https://gitlab.com/second-bureau/pegasus/recup

Some Refactoring will happen, so now i need tests before big moves. Tests are security for big plans.

So this issue is :

• set up mocha tests ,
• then i do architecture refactoring as specified below
• end of refactoring :
  • will add karma jasmine tests, fully typescripted angularized client, ready for electronjs
  • will have REST API Endpoints matching every backend service implemented in express full typescripted, RxJS, mongoose.

So Ok, Here i a little architecture/product roadmap :+1: , where i'll split server into multiple micro-services (having David Pageot's execellent presentation of K8S Istio in 2018 devoxx) :

• secrets manager : you can give the secret omega needs to persist your template to your github repo. You load from a repo, initially, but you can use antoher to persist your work (add, commit n push). That secret can be managed trough the secret manager, where you can set modify delete your secret, be ti a private key, or an app API token. Copy pasting that secret inside a beautiful angular/bootstrap interface, like in openstack. Then why not bring integration with HashiCorp Vuault, to even manage secret rotation, with expiry. All in all, since there 's secret transportation involved here :
• https mandatory, with hashicorp vault free ipa lets encrypt workflow guide

• service 1 (workspace persistence service) : he knows how to rebase href and src attributes properly inside HTML streams. He gets the help of things like Parcel, that helps him test whether any nestested-nested-nested-nested frontend dependency is missing or broken. He knows too, how to do the exact same opposite thing :

  • So service1 is called to load an html File , either because its loading the default index.html at initial opening of the template, or because clicked in the treeview by the user,
  • at this point, service 1 knows the path, relatively to the workspace/omega folder, of the html file to load in editor. the path is workspace/omega/path/to/the/page/file/to/load/nameoffile.html
  • service 1 reads trhe HTML inside workspace/omega/path/to/the/page/file/to/load/nameoffile.html, and tansforms it before passing it back to the grapesjs client.
  • service 1 transforms the HTML stream , with those rules (mocha tests for those rules) :
  • for every href and src tag attribute in the HTML stream, service1 parses its value and, if the value starts with a double slash //, or with one of the strings ftp://``ftps://, htttp://, and htttps://, then the value is not changed. In all other cases, the string omega/ is prepended to the value. That is because Omega defines a static route to folder workspace .
  • To do the opposite thing before saving, alll service 1 has to do, is to reset back the old values, using one thing i did not mention : for every tag omega has changed an attribute value, omega added an HTML compliant attribute data- prefixed, namely data-omega-loader-src (or data-omega-loader-href, in case it was href that was changed), which has a value , the old value of the src or href attribute. Yeah. Great like that i can really easily unload, just have to get the old value and reset the attribute, plus removing the data-omega-loader-{src, href}, from the HTML, though it wouldn't hurt to leave it there...No we'll have to be able to persist it ti git, because it's dev, like we version the test classes with code, but we don't send the unit test classses to production, inside artefacts;
  • So once the HTML processed, it is sent to the GrapeJS client, exactly to the StorageManager, and its load method, wich request the HTML from serveur, providing a file path, relative to the workspace/omega folder.
  • Ok i understand one thing : the GgrpesJS component must not be initilized in a page, before the user has clicked on the first file to load for editing. So the user must first, visualize the content of the workspace/omega folder, as a treeview. Then the user clicks on an HTML file, and then only the GrapesJS omponent is initialized in the page. Like I could insert it with jQuery.. I'll do that first, and see if the iitial load method of the Storage MAnager is stilll called, while I injet the editor with jQuery
  • now the HTML is sent to cleint with a JSON objet of the form :

  {
      html: 'the HTML code of the page to load, afterit was processed for rebasing href and src tag attributes'
  }

  • Before loading the new page for edition, the previous is persisted to the workspace/omega folder. To do that, service 1 gets the HTML from the GrapesJS client, more accurately from my implementation of the GrapesJS Editor Storage Manager, as a JSON Object containing two strngs, style and html , basically the edited content.
  • service 1 processes the html string, to restore the href and src tag attributes values to what they were before the loading the HTML to the GrapesJS Editor. Only then the "cleared' HTML is persisted back to the file it came from in workspace/omega/some/sub/folder. The style string is appended to a file (created if does not exist) named workpsace/omega/some/sub/folder/omega-editon.css, and its <link href="omega-edition.css" ref="style"> is added inside the HTML page.
  • service 1 receives from the client a JSON object, exactly from my implementation of the Storage Manager ), with the following structure :

{
  html: `<div>...</div>`,
  css: null,
  style: '.someclassbasedrule { background-color: #FEFEFE }  ; div section h2 {font-family: "Montserrat", sans-serif;}',
}

• service 2 : he knows how to load and save a template from / to a given github.com/gitlab.com/ any git repo. from git cloning the git repo, you can then choose to commit n push to the same repo, or start working on a new template you will persist in a different git repo. About That marketplace, see On the Socialized Marketplace concept below.

About that structure, see for example :

• https://grapesjs.com/docs/modules/Storage.html#storage-api
• https://grapesjs.com/docs/api/editor.html#getstyle
• https://grapesjs.com/docs/modules/Storage.html

My GrapesJS Storage Manager

On the official GrapsJS documentation, You wll read in the Storage Manager guide :

const LandingPage = {
  html: `<div>...</div>`,
  css: null,
  components: null,
  style: null,
};
// see https://grapesjs.com/docs/modules/Storage.html#store-and-load-templates

What you have to understand here, is that the HTML the GrapesJs editor user is working on, is what the GrapesJS author calls The Canvas. And that the canvas is all set by three properties, namely html, css, style. I know it because I tested it my self, wih release 0.0.x, x being 1, 2, 3 4, 5, 6, and 7.

With seven things, I end up here , deciding that I will develop my own implementation of a "Storage Manager", and that it will send a request to a backend REST API (API that I will implement too), request with a JSON object

That REST API endpoint will be service 1, it will know how to persist a template modification persistance request, sent with a JSON Object as the payload to persist. The edited HTML/CSS will be persisted :

• As for the HTML , inside the file, inside the workspace/omega folder, the HTML originally came from.
• As for the CSS, inside the HTML fle folder, as a omega-edition.css file, persisting the style string

[Jean-Baptiste-Lasselle]

i'll probably use a streaming processing model like jacksons' in java for XMLand JSON, to parse the HTML in an event driven way, relyng on the "marks" left by the attributes data-omega-loader-href and data-omega-loader-src on tags. That way, we do only streaming on parsing HTML / XML (the mechnaism is extensible to MJML), instead of trying to remembervalues with such things as pmongo db, or even just a big JSON kept in RAM by NodeJS inside the os process context.

I found https://www.npmjs.com/package/parse5-sax-parser , which is both purposed specifically for HTML, not just XML (loike the one i'm currently using), and is a SAX style parser, based on event emission

[Jean-Baptiste-Lasselle]

Workflow

Actors

• the user of Omega

Initial Context

• the user has found an HTML 5 template on the web
• the user has created a github repo of URI $TEMPLATE_GIT_REPO_URI
• the user has versioned the template he found somewhere, inside the git repo, complying qith the following rules, the dev teams imposed him to approve his git repo access to an omega template repository.

• an omega template repository simply is a git repo, containing only two files at its root :

  • a json file named omega.json : this json object must have a given structure :

    
    {
    templates : {
    'template1' : { /* The unique omega repository id of your template */
    title: 'A great starter template for our usual e-commerce customers websites ',
    git_ssh: 'git://github.com:Jean-Baptiste-Lasselle/grapejs-server',
    git_http: null /*  should be used for BOTH http AND https starting uris of git repositories */ 
    },
    'template2' : { /* The unique omega repository id of your template */
    title: "Our starter template for typical software product's github pages ",
    git_ssh: 'git://github.com:Jean-Baptiste-Lasselle/grapejs-server',
    git_http: null /*  should be used for BOTH http AND https starting uris of git repositories */ 
    }

  } }

  
  each pair a set of git repos, that the dev team guarantees they are compatible with omega  
  * 

[Jean-Baptiste-Lasselle]

then :

• omega tries to load the template from a new repo, and that new repo will be added to the omega template registry if and only if the omega user validates he could properly work with it.
• So in an omega registry, every entry stands for a user validation, against a given version number of omega, so that the user managing the omega registry, has to sign his addition with a difgital signature used behind with git for signing commits and merge. Eventually, to make an omega registry completely independent of any git concept, I mean that its the omega.json file itself, that should be digitaly siigned, and I need to provide standard way with automation, to check signature.

How are signature checked?

• something public sould be advertised like download here this thing to verify my digital signature ...
• something that has to be made available by the signer, or a certificate authority : I'll have to check that workflow so that I know if I have to add metadata fields regarding signing and signature checking processes.

So all in all a registry is something people give you a gaurantee about : "Yes this thing that I provide, does workk properly with that other thing"