[RFC] nativefier-catalog and interactivity

matthewruzzi commented 3 years ago

Nativefier catalog is a catalog of configs for nativefier apps. It currently stores app configs as a json file with the following structure:

{
  "app-token": {
    "name": "App Name",
    "targetUrl": "https://appurl.example/",
    "counter": true,
    "bounce": true
  }
}

The options are passed directly to the nativefier Programmatic API. The UI is built using enquirer autocomplete prompts and shows a searchable list of available app configs.

screen recording showing nativefier-catalog in hyper terminal

I also could make a graphical way of viewing it to go along with my nativefier-gui project.

The json file is curently stored in the same repo as the code but should probably be moved to a different repo to seperate app adition prs from code pr like how Homebrew/brew (package manager code) is separate from Homebrew/homebrew-core (package repo). Having a repo of just configs could allow multiple different clients to all access it and share the same configs. (the default nativefier cli (see question 3), nativefier-catalog cli, nativefier-gui, something else?... )

Questions

Should this integrate with the existing nativefier-icons repo somehow?
Should there be a folder for userscripts to go along with apps.json or should they be linked as gist urls?
Should there be some kind of url (regex?) matching so the configs are used by default on the standard nativefier cli? (eg. use counter by default on gmail because the gmail entry in apps.json has counter enabled.

Any other ideas? Comments? Questions? Different directions?

@ronjouch @TheCleric

ronjouch commented 3 years ago

@mattruzzi awesome! Following up soon with my thoughts.

ronjouch commented 3 years ago

@mattruzzi @TheCleric thanks! Here are my thoughts. Before you read, I want to stress out that the size of this brain dump shouldn't make it overwhelming/intimidating. I just happen to have seen this idea bounced a lot, so naturally I have a few ideas. I have zero expectations about this, and me suggesting something doesn't mean I'd expect it to be here yesterday on a first release. Alright, here it goes:

Client UX

Your interactive prompt with a list & autocomplete looks rad, and would help a lot making Nativefier easier to use for people not familiar with the command-line.

I don't think your new interactive UI obviates the need for current Nativefier; some users will want control over specific flags, add custom --injects, etc. Use cases I can picture:

Users could want to extend a catalog config: Will build Google Docs. Are there extra flags you would like to add?
On users launching a regular Nativefier build with their flags, we could auto-detect that the domain matches a catalog domain, and offer to use its config: You requested to build an app on mail.google.com. Nativefier-catalog has a config for this site, named "Gmail", with an icon, options [counter, bounce], and injectable CSS+JS. Do you want to use it? . This "do you want to use it" could then be selective: a user could opt in to just the icon, or icon + one JS but no custom CSS, etc.

This I think lets us answer the question "Where should this client live? Should it keep being a wrapper (like your current project is, @mattruzzi), or should the client & logic live inside Nativefier proper?". My thinking about this is twofold:

Super extra happy to see you experiment in your wrapper for now.
But later on, given the use cases listed above (extending, noticing a manual build that could be helped by a config) and the need for extra interactivity in core Nativefier that your project surfaces, I think we will want this in Nativefier proper. A wrapper won't do.
- Also, if/when the interactive wrapper matures, we'll want to publicize it to users. To do that and for it to be the default experience, it should belong to main Nativefier. Leaving the wrapper outside would be like "hey there's this better easier thing over here, go use it", to which a frank new user would reply "if it's better, why isn't it the default?"

Configs repository

I agree that the json file should live in its own repo, to let it be its own thing, with maybe its own maintainers helping digest quickly community contributions if the volume increases, and yes it opens the way to other clients.

CSS & JS injects ("userscripts") must live alongside in the repo, yes. No gist URLs, especially since we'd be giving these userscripts more visibility and accessibility, they should live in the repo, for security.

This repo should absolutely replace the severely-uncared-for nativefier-icons repo and its gitcloud client, yes. These things are two of the weirdest things in Nativefier, leftovers of Jia (Nativefier creator) doing dubious things as novice dev, generating HTML using Jekyll, that is later on parsed in gitcloud (using html-parsing npm dep cheerio) to grab icons, rube-goldberging the act of fetching data, that would be a simple JSON.parse if it were JSON to begin with. It's nonsensical and Jia himself jokes about it 😄. Yes, let's merrily burn this and replace it all with a simple JSON. While doing this, we should merge the mountain of icons accumulating in open PRs and that I'm ignoring 😬.

The new repo of configs could be entirely JS-less / only a JSON (plus CSS+JS injects). Contributors would submit PRs modifying the JSON (it's reasonable, JSON is readable) and potential css/js injects. We'd have GitHub Actions-based CI that does sanity checks, for maximally automatic checks and minimal PR-reviewing overhead:

Validate the JSON schema is respected
Validate the JSON keys are sorted
Validate that css/js injects referenced in the JSON exist in the tree
Require js & css inject to have a MIT license header
Check & lint: js injects (with eslint) and css injects (with stylus)
Crush PNG icons (with pngquant)

Schema

In your RFC, you indexed the JSON by an "app key/token". My initial thinking was to index by domain, but I may be wrong. Domain/subdomain stuff might be tricky, change with time, so maybe it's not a good idea and you're right with an arbitrary key. Maybe this key should be the app name shown to the user, though? It's totally fine to have any character and spaces in a JSON key.

Then, I was imagining something like what you're suggesting:

url
options (with nested options like the counter and bounce you already mention, but any other options should be usable here)

Options should probably not be raw key: value couples, we'll definitely want an object per option, letting us add extra metadata about each option. I imagine we could want each option to have:

A required default / recommended flag, conditioning whether this is used by default. This would let users contribute flags/injects that we know are not desirable for everybody, but are still nice to offer, ready for cherry-picking in interactive mode.
An optional description field, helping explain to users what some options do (e.g. if WhatsApp has 6 JS injects offered, we want to let users know what each of them does)

Misc

As experimentation happens in your interactive client, please keep an eye on npm dependencies, and align them with Nativefier's. If picking a dep, review its activity, popularity, and sub-dependencies weight; I want to keep Nativefier dependencies weight minimal. I'm not putting a hard no on extra deps, and for example the interactive prompt deserves a dep and enquirer looks like a sane choice, but for example I'd ask you replace meow with commander we already use.

matthewruzzi commented 3 years ago

# Todo List: - [ ] Move apps.json to separate repo - [ ] Implement new schema - [ ] Migrate icons from nativefier-icons ... @ronjouch feel free to edit.

matthewruzzi commented 3 years ago

For adding some new features (eg. auto detecting and using settings based on a url or customizing the settings used) I'm not quite sure where to start. I could add the ability to customize args or pass a url to this wrapper but if it eventually gets integrated into nativefier it would be awkward to have a wrapper for nativefier as a part of nativefier. I could try to add code to auto detect urls and use settings from the catalog into my fork the main nativefier repo (I think I would put the code into https://github.com/nativefier/nativefier/blob/master/src/cli.ts) but I'm not sure exactly what to do.

matthewruzzi commented 3 years ago

I'm working on a adding a way of specifying css/js injects in nactivefier-catalog but I am having problems related to #458 when passing multiple files to inject where it seems to overwrite it each time I pass it a --inject so only the last one is actually included.

ronjouch commented 3 years ago

Follow-up of https://github.com/nativefier/nativefier/issues/1147 : udemy.com is a case where the catalog should suggest using our flag --widevine

TheCleric commented 3 years ago

@ronjouch did you mean to close this?

ronjouch commented 3 years ago

@ronjouch did you mean to close this?

Whoopsie 😄. No! Just meant to add the link to #1147. Thanks.

matthewruzzi commented 3 years ago

I was thinking for every site that is in nativefier-catalog we also could put it in a markdown file with explanations for all of the flags used so that the json schema for nativefier-catalog can remain clean and not have the explanations for the flags in it.

Edit: actually I changed my mind I think we just need a whole new schema that includes proper support for explanations See https://github.com/nativefier/nativefier/issues/1130#issuecomment-830675601

matthewruzzi commented 3 years ago

Eventually when this gets combined with the main nativefier codebase maybe we could make it so that if you ran nativefier without passing a site or any flags it could enter an interactive mode asking you to choose from an existing site or build an app for your own with an interactive wizard asking you questions eg.

$ nativefier 
(we could have some way of asking if you want to create your own custom config for the app or if you want to use a premade one here, or we could default to the custom app maker if the user types an app that is not in the catalog})
(custom app maker:)
? What is the url to that website you want to nativefy … https://mail.google.com/
? What name should the app be saved as … Gmail
? Do you want a counter (insert explanation of counter?) (y/n) … y
? Do you want to use disk cache (insert other explanation) (y/n) … n

ronjouch commented 3 years ago

I was thinking for every site that is in nativefier-catalog we also could put it in a markdown file with explanations for all of the flags used so that the json schema for nativefier-catalog can remain clean and not have the explanations for the flags in it.

@mattruzzi I disagree: I don't see anything dirty about having a comment key for each catalog entry. I'd prefer it to an explosion of small markdown files. Everything remains closely at the same place, obvious to understand & maintain without having to go fish for a separate file.

To which someone may argue "But in a md file we'll have space, access to formatting, and multiline". I'm not sure we really need these, and being "cramped" in a JSON file will push us to remain concise, and that's good.

Worse is better.

matthewruzzi commented 3 years ago

In the current version the json is directly passed to nativefier. I am working on adding injects to it and have added the following to the schema:

"example": {
    "name": "Example",
    "targetUrl": "https://example.com/",
    "injects": {
      "does something ": {
        "path": "./injects/example/do-thing1.js",
        "desc": "does something "
      },
      "does something else": {
        "path": "./injects/example/do-thing2.js",
        "desc": "does something else"
      },
      "important fix": {
        "path": "./injects/example/fix-thing3.js",
        "desc": "fix something else"
      }
    }
  }

I am using the following code prompt for the userscripts and remove the unused ones in memory at runtime:

if (apps[name].hasOwnProperty("injects")) {
        var injectsList = Object.keys(apps[name].injects);
        console.log(injectsList)
        const prompt = new MultiSelect({
            name: 'injects',
            message: 'Available userscripts/stylesheets',
            hint: '(Use <space> to select, <return> to submit)',
            choices: injectsList
        });

        prompt.run()
            .then(answer => runNativefier( getOptions(name, answer)))
            .catch(console.error);
    }}

function getOptions(name = {}, injects = {}) {
        var toInject = [];
        injects.forEach(element => {
            toInject.push(apps[name].injects[element].path)
        });
        var  options = apps[name];
        delete options.injects; // delete the list of available injects (see json above)
        options.inject = toInject; // add the only ones that were selected in the prompt to the options passed to nativefier
        console.dir(options);
        return options;
    }

function runNativefier(config = {}) {
    nativefier(config, function (error, appPath) {
        if (error) {
            console.error(error);
            return;
        }
        console.log("App has been nativefied to", appPath);
    });
}

The current way I am adding extra features to the schema is by adding code that will translate them to standard nativefier configs at runtime (see above where I translate "injects" (my schema addon for injects) to "inject" (a nativefier flag).

I don't know how scalable this is for adding more options/descriptions for options to the schema.

matthewruzzi commented 3 years ago

What do you think of the following schema? Do you have any different ideas?

"example": {
    "App name": {"name": "Example"},
        "App Url": {"targetUrl": "https://example.com/"},
    "Fix something with inject": {"inject": "injects/example/fix-thing3.js"},
    "Do something else with inject": {"inject": "injects/example/do-thing2.js"},
    "Use a Widevine-enabled version of Electron to allow video playback":{"widevine":true},
    "Explanation": {"flag":"value"}
  }

We could provide some kind of interactive prompt that would allow users to toggle/edit each option.

ronjouch commented 3 years ago

What do you think of the following schema? Do you have any different ideas?

@mattruzzi I like the new schema you're proposing:

I like that it's "solution-oriented": you offer a "bag of solutions" ("Fix something with inject", "Use Widevine ...", etc), each customizing one or several options (in your examples, all the "solutions" override a single flag, but we could imagine an option/solution setting e.g. inject, and widevine, and userAgent, if they're all needed cohesively to enable something) and cherry-pickable. I think that's great.
- To follow-up on our "explaining why" discussion above, each cherry-pickable solution/option could have an optional "comment" or "catalogComment" key, that is used by the interactive UI to explain the solution in detail and e.g. link to discussion in an issue, if users ask for details. This key would be of course stripped when passing flags to Nativefier.
For schema validation, what about nesting the solutions inside a "solutions" (or "options") root key. If we don't do such nesting, we'll have to allow arbitrary keys at the root, and I don't like that as I'd like to be able to validate required keys like "name" and "url" (see next comment).
I find strange to see required "App name" and "App Url" spelled like the rest of the "bag of cherry-pickable options". These two (and maybe more) aren't like the others, in that they're mandatory. Feels like the keys should be standard to "name" and "url", to offer a consistent experience and to do a good validation job. But I like that they use the same "it's just a define/override" approach.
One question: how/where will search work? I had in mind to search in [ catalog keys ("example"), name ("Example"), andurl("https://example.com/"`). Makes sense?

We could provide some kind of interactive prompt that would allow users to toggle/edit each option.

Yeah, that fits nicely with the workflow you're suggesting.

matthewruzzi commented 3 years ago

"example": {
    "name": "Example",
     "targetUrl": "https://example.com/",
     "additionalOptions": {
         "Allow video playback": {
            "flags": {"inject": "injects/example/fix-video.js", "widevine":true},
            "default": true, 
            "comment":"Longer explanation"},
            "linktomoreinfo?" : "https://explanation.example/something"
}
  }

Should the link be part of the comment or a different field? Maybe we could standardize it and have a GitHub issue for each one?

matthewruzzi commented 3 years ago

@ronjouch I just committed some Proof of concept code for the new schema.

https://github.com/mattruzzi/nativefier-catalog

I'm starting to see a lot of duplication in it as a lot of apps use the same fix (all Google apps need user agent fix). Should we add a section for common fixes that can be referenced from multiple apps?

ronjouch commented 3 years ago

I'm starting to see a lot of duplication in it as a lot of apps use the same fix (all Google apps need user agent fix). Should we add a section for common fixes that can be referenced from multiple apps?

@mattruzzi yeah indeed, a section for common fixes (that are reference-able-by-name by apps) seems legit. So:

At the root of the json there could be a "__commonOptions" key containing an object of the common options, named e.g. "useragent-firefox" in your case.
Then, next to additionalOptions we'd have something like importedOptions (or referencedOptions ? or libraryOptions ? , and in your case it would be [{ "libraryOptionName": "useragent-firefox", "default": true }] . Something like this?

matthewruzzi commented 2 years ago

@ronjouch @TheCleric I noticed there is another very similar project to both nativefier-catalog and nativefier-gui. https://github.com/vinifmor/bauh#native-web-applications https://github.com/vinifmor/bauh-files/blob/master/web/suggestions.yml https://github.com/vinifmor/bauh-files/tree/master/web

I don't know if this changes anything.

Is there any reason switch to yaml instead of json?
Could we collaborate with them and share the catalog of apps and fixes?

ronjouch commented 2 years ago

Could we collaborate with them and share the catalog of apps and fixes?

@mattruzzi two thoughts:

Yeah, seems legit to want to collaborate
At the same time, I see a few objections to trying hard to re-use their stuff.
1. It's not as if their suggestions.yml had hundreds of apps and important stuff. It's mostly naming/localization and minor things.
2. They seem to re-implement icon handling logic (for good reason as I/nobody maintains our nativefier-icons repo) and use icons from the web
3. Many of the default options they set are debatable. As discussed above, it feels preferable to only set the minimal amount of options to make a site work at all (e.g. widevine, inject, useragent, etc) but let cosmetic (maximized, start in tray, etc) things up to everybody's taste, maybe offering it but not defaulting to it.

Sooo, if I were in your shoes I'd err on the side of doing my thing, keeping in mind the features they support, and then later inviting them to contribute.

Is there any reason switch to yaml instead of json?

I was suggesting json initially for absolute maximal platform (node/js) platform friendliness at the detriment of maintainer friendliness (see HACKING.md / point 4. about deps). That being said, for such a database that is bound to be frequently modified by humans, it's probably reasonable to use YAML, as it's much more readable. Also, I see that yaml and js-yaml both seem well-maintained and are very light on dependencies (0 for one, 1 for the other). So, no objection to YAML.

matthewruzzi commented 2 years ago

If we use yaml, should we require keys with spaces to be quoted? It isn't required by yaml, but I don't know if there is any reason we would want to do it. For things like Change user agent to prevent Google from blocking access, instead of having them in this file as keys, we might want to move them to a different file and give them simpler names to reference by like google-ua-fix or something, and then add a key like referencedOptions: [ google-ua-fix ]

I used an online json to yaml converter and got this:

gmail:
  name: Gmail
  targetUrl: https://mail.google.com/
  additionalOptions:
    Show counter on app icon in dock:
      flags:
        counter: true
        bounce: true
      default: true
      comment: (macOS only) Use a counter that persists even with window focus for
        the application badge for sites that use an "(X)" format counter in the page
        title (i.e. Gmail).
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
google-meet:
  name: Google Meet
  targetUrl: https://meet.google.com/
  additionalOptions:
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
google-docs:
  name: Google Docs
  targetUrl: https://docs.google.com/
  additionalOptions:
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
google-sheets:
  name: Google Sheets
  targetUrl: https://sheets.google.com/
  additionalOptions:
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
google-slides:
  name: Google Slides
  targetUrl: https://slides.google.com/
  additionalOptions:
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
  example2:
    name: Example2
    targetUrl: https://example.net/
    additionalOptions:
      Allow video playback:
        flags:
          inject: injects/example/fix-video.js
          widevine: true
        default: true
        comment: Longer explanation

@ronjouch Should I use the .yml or .yaml extension?

ronjouch commented 2 years ago

@mattruzzi

If we use yaml, should we require keys with spaces to be quoted? It isn't required by yaml, but I don't know if there is any reason we would want to do it.

Quoting not necessary indeed, and given the main goal of YAML is to be human-friendly, I'd err on the side of omitting them.

For things like Change user agent to prevent Google from blocking access, instead of having them in this file as keys, we might want to move them to a different file and give them simpler names to reference by like google-ua-fix or something, and then add a key like referencedOptions: [ google-ua-fix ]

👍, I like the idea of a "named options referrable to" section, containing common fixes that we can refer to by name.

Should I use the .yml or .yaml extension?

Dunno; I don't see any objection to one or the other.

matthewruzzi commented 2 years ago

@ronjouch I noticed another similar project made by @Nickersoft. I don't know if that changes anything about our plans. https://github.com/Nickersoft/hop

matthewruzzi commented 2 years ago

@ronjouch Wow it's been a year! I just pushed code that switches the apps file from json to yaml. We could use yaml anchors to fix the duplication. I switched the Google user agent fix to use them. https://www.educative.io/blog/advanced-yaml-syntax-cheatsheet#anchors

gmail:
  name: Gmail
  targetUrl: https://mail.google.com/
  additionalOptions:
    Show counter on app icon in dock:
      flags:
        counter: true
        bounce: true
      default: true
      comment: (macOS only) Use a counter that persists even with window focus for
        the application badge for sites that use an "(X)" format counter in the page
        title (i.e. Gmail).
    Change user agent to prevent Google from blocking access: &google-ua-fix
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
google-meet:
  name: Google Meet
  targetUrl: https://meet.google.com/
  additionalOptions:
    Change user agent to prevent Google from blocking access:
      flags:
        user-agent: firefox
      default: true
      comment: 'Note: lying about the User Agent is required, else Google will notice
        your "Chrome" isn''t a real Chrome, and will refuse access.'
google-docs:
  name: Google Docs
  targetUrl: https://docs.google.com/
  additionalOptions:
    *google-ua-fix
google-sheets:
  name: Google Sheets
  targetUrl: https://sheets.google.com/
  additionalOptions:
    *google-ua-fix
google-slides:
  name: Google Slides
  targetUrl: https://slides.google.com/
  additionalOptions:
    *google-ua-fix
example2:
  name: Example2
  targetUrl: https://example.net/
  additionalOptions:
    Allow video playback:
      flags:
        inject: injects/example/fix-video.js
        widevine: true
      default: true
      comment: Longer explanation

nativefier / nativefier