Plugins!

[terreng]

Plugins would allow anyone to create additional features or customize the behavior of the app. We could have a list of additional plugins on the website. I think a plugins system would strike a nice balance between keeping the app simple, but also letting people take it further if they want additional functionality.

Plugins would consist of a config file and some javascript. The config file would have details like the plugin name and version, and an array of additional options (that could be checkboxes, textboxes, or number inputs). In the UI, it would add an extra options section for each plugin with these options. Then in the javascript, you could write a custom script that can apply custom logic to requests, like adding headers or turning it into a proxy. This would work like the custom scripts/custom request handler features you've already made - it would have access to req and res, as well as the values of all the server options.

Users could download plugins - possibly as a zip of multiple files or perhaps as one combined file with a custom file extension - and then add them into the app from the UI or by placing them in a directory within the app's data folder like /plugins.

We'd want to make sure that it's easy to write basic plugins for things like adding additional headers. Ideally, this should be possible with only a few lines of code.

Alongside the introduction of plugins, I'd like you to create the first official plugin which adds the proxy feature.

Let me know what you think!

Here are some prototype/example screenshots of what it might look like:

[ethanaobrien]

Oh, I missed that you already had written importPlugin and removePlugin. My bad.

It's all good

It would be great if you could update them however needed so that adding and removing plugins takes effect immediately.

When a plug-in is removed/added, the server the plug-in is on will need to be restarted. I'll add a function that it calls when these things happen that will set the global plugins variable. Could you write the part that restarts only the needed servers. I saw you managed to do this when settings were modified

Also, could you use fs.watch to watch for changes to the plugin directory, and reload the plugins when anything changes? This would enable someone to edit the plugin's javascript and have the changes take effect in real time, which would be cool.

We can do this, although if the main plug-in config file was modified, the server that the plug-in is attached to would need to restart

[ethanaobrien]

Also, how will fs.watch work with macos security bookmarks?

[terreng]

Could you write the part that restarts only the needed servers. I saw you managed to do this when settings were modified

Sure. I assume that it should restart any servers that the plugin is enabled on? If so I'll write a function like restartServersWithPlugin(pluginid). Does that work?

Also, how will fs.watch work with macos security bookmarks?

The app always has permission to access the contents of its own app data directory, so security scoped bookmarks don't matter. This will only become a problem if we want to use fs.watch on directories outside, like a folder the user has selected.

[ethanaobrien]

Sure. I assume that it should restart any servers that the plugin is enabled on? If so I'll write a function like restartServersWithPlugin(pluginid). Does that work?

Yes, this would work.

The app always has permission to access the contents of its own app data directory, so security scoped bookmarks don't matter. This will only become a problem if we want to use fs.watch on directories outside, like a folder the user has selected.

Ok good. Are we going to include in the documentation that for development purposes you can actively edit in the plugins folder?

[terreng]

Ok good. Are we going to include in the documentation that for development purposes you can actively edit in the plugins folder?

Sure. Speaking of documentation, we'll need to document the plugin.json format and javascript handlers you can write. Would you like to take on writing the documentation for plugins? Or we can do it together.

[terreng]

I added a function restartServersWithPlugins(pluginids) to the main branch. It takes in an array of plugin ids.

[ethanaobrien]

Would you like to take on writing the documentation for plugins? Or we can do it together.

I'd like to try to write it, and then have you revise as needed, if that's ok with you

I may not be able to work on any of this until next week, I've been really busy with work. Sorry about that

[terreng]

I'd like to try to write it, and then have you revise as needed, if that's ok with you

Sounds good.

I may not be able to work on any of this until next week, I've been really busy with work. Sorry about that

No worries!

[terreng]

@ethanaobrien I've made some changes to the plugin code.

• I added a function validatePluginManifest(manifest) which checks to make sure that the manifest is all valid, including the options, to make sure it doesn't crash the UI. I also added this check before plugins are loaded in.
• I removed the activate and deActivate functions because we won't need them.
• I added a check to make sure that the folder name of plugins matches their id when loading them in. It looks like we're assuming everywhere that the ID of a plugin matches the name of the folder it's in, so we should enforce that wherever possible.
• In index.js, I renamed the global variables so that plugin refers to the import and plugins refers to the list of plugins.

All of these changes are on the plugins-darkmode-reorder branch, so you should probably work from that instead of main.

I came across an error that will only happen on macOS, which is related to the hidden .DS_Store file that macOS uses:

could not import plugin .DS_Store Error: Entered path is not directory
    at new FileSystem (/Users/terren/Documents/GitHub/web-server/WSC/FileSystem.js:223:19)
    at getPluginInfo (/Users/terren/Documents/GitHub/web-server/plugin.js:91:14)
    at Object.getInstalledPlugins (/Users/terren/Documents/GitHub/web-server/plugin.js:229:30)
    at Object.<anonymous> (/Users/terren/Documents/GitHub/web-server/index.js:28:25)

Could you make sure to ignore dot files in the plugins directory instead of erroring?

[terreng]

I got started writing documentation for plugins. I split it into three parts: Introduction to plugins, Plugin manifest file, and Plugin script. It would be great if you could write the Plugin script page to explain how scripts work. I wrote down some ideas you should make sure to include. The file is located at website/src/docs/plugin script.md.

[terreng]

@ethanaobrien I went ahead and implemented live reloading of plugins using fs.watch and the restartServersWithPlugins function I wrote.

The plugins UI is all done. And plugins overall are pretty much all done.

I think the only thing left for you to do is fix the .DS_Store issue I mentioned above, and write some documentation.

Oh and also, make sure to clean up / make a new repo for the Proxy plugin, as we talked about. And two changes you should make to plugin.json:

• Because forceSite is a string, the value for default should be an empty string instead of false. The plugin manifest validation I wrote will reject this unless you fix it.
• For each option, you can add a description field to explain what it does, which will show up in the UI as a question mark you can click for more info.

[terreng]

Please go ahead and test everything on the plugins-darkmode-reorder branch and let me know if you find any bugs!

[ethanaobrien]

I went ahead and implemented live reloading of plugins using fs.watch and the restartServersWithPlugins function I wrote.

Thank you, sorry for my inactivity. I've had a lot going on.

I think the only thing left for you to do is fix the .DS_Store issue I mentioned

This should be easy. I'll do this as soon as I can

and write some documentation

I'll do this when I can

and also, make sure to clean up / make a new repo for the Proxy plugin

I actually determined that I'm going to keep it in the same repository, since the only difference would be that one has a plugin.json file and one doesn't (since the project itself is based as a plug-in). As for the proxy project itself, I do need to clean up and write the documentation

I should be able to test everything next week!

[terreng]

Sounds good! Once you're done with all that we should be ready to do a release.

[terreng]

One more issue I found. Importing plugins from ZIP files isn't working. Here's the error:

(node:28229) UnhandledPromiseRejectionWarning: TypeError: Cannot read properties of null (reading 'async')
    at /Users/terren/Documents/GitHub/web-server/plugin.js:157:70

This line:

const manifest = JSON.parse(await zip.file('plugin.json').async("string"));

Could you take a look and fix it?

[ethanaobrien]

This is happening because it doesn't like how the zip is zipped. Everything at the moment needs to be in the base zip directory (when you open the zip file, the file should be there, no folders) there's no incredibly efficient way around this, although if no plugin.json was found we could check the first folder found. What do you think?

[terreng]

Everything at the moment needs to be in the base zip directory (when you open the zip file, the file should be there, no folders) there's no incredibly efficient way around this, although if no plugin.json was found we could check the first folder found.

It would be great if we could make that work, because I can definitely imagine people making this mistake.

I'm still unable to import from a ZIP even when it's formatted correctly. There isn't anything in the terminal, but it's still erroring.

Here are the files I'm testing with. It works when I pick a folder with the files, but not when I pick the zip file.

Archive.zip

plugin.json

```json { "id": "my_example", "name": "Example\n\nPlugin", "script": "script.js", "options": [ { "id": "my_checkbox", "name": "Example\n\ncheckbox", "description": "Enable if you like checkboxes.", "type": "bool", "default": false }, { "id": "my_textbox", "name": "Example\n\ntextbox", "description": "Enter your favorite word.", "type": "string", "default": "" }, { "id": "my_number", "name": "Example\n\nnumber input", "description": "Specify your favorite number.", "type": "number", "default": 50, "min": 0, "max": 100 }, { "id": "my_dropdown", "name": "Example\n\ndropdown", "description": "Choose your preferred color.", "type": "select", "default": "red", "choices": [ { "id": "red", "name": "Re\n\nd" }, { "id": "yellow", "name": "Ye\n\nllow" }, { "id": "green", "name": "Gre\n\nen" }, { "id": "blue", "name": "Blu\n\ne" } ] } ] } ```

script.js

```javascript function onStart(server, options) { // Do nothing } function onRequest(req, res, options, preventDefault) { if (options.header == true) { res.setHeader('my-header', 'hello world'); } } module.exports = {onStart, onRequest}; ```

Happy Holidays by the way :)

[ethanaobrien]

Very sorry for the delay. I just pushed a change to the plugins-darkmode-reorder branch that should fix this issue. I also added some comments in the code of some bugs I found. I was also able to load the example plugin you provided and ran it (I had to modify it, because options.header returned undefined. I just set the header, no conditions, and it works)

Let me know if there are any more plugin problems!

Happy new years!

[terreng]

Thanks for the fix! I tried adding a plugin that was zipped improperly (folder inside the zip), and now I'm getting these errors:

copy /Users/terren/Library/Application Support/Simple Web Server/plugins/my_example
(node:43550) UnhandledPromiseRejectionWarning: TypeError: Cannot read properties of undefined (reading 'startsWith')
    at getZipFiles (/Users/terren/Documents/GitHub/web-server/plugin.js:101:28)
    at copyFolderRecursiveSyncFromZip (/Users/terren/Documents/GitHub/web-server/plugin.js:111:17)
    at /Users/terren/Documents/GitHub/web-server/plugin.js:156:19
could not import plugin .DS_Store Error: Entered path is not directory
    at new FileSystem (/Users/terren/Documents/GitHub/web-server/WSC/FileSystem.js:223:19)
    at getPluginInfo (/Users/terren/Documents/GitHub/web-server/plugin.js:91:14)
    at Object.getInstalledPlugins (/Users/terren/Documents/GitHub/web-server/plugin.js:245:30)
    at Timeout._onTimeout (/Users/terren/Documents/GitHub/web-server/index.js:210:102)
    at listOnTimeout (node:internal/timers:559:17)
    at process.processTimers (node:internal/timers:502:7)
Error stating "/Users/terren/Library/Application Support/Simple Web Server/plugins/my_example/plugin.json" Error: ENOENT: no such file or directory, stat '/Users/terren/Library/Application Support/Simple Web Server/plugins/my_example/plugin.json'
    at statSync (node:fs:1588:3)
    at t.statSync (node:electron/js2c/asar_bundle:2:4510)
    at getByPath.getFile (/Users/terren/Documents/GitHub/web-server/WSC/FileSystem.js:40:24)
    at FileSystem.getByPath (/Users/terren/Documents/GitHub/web-server/WSC/FileSystem.js:231:24)
    at getPluginInfo (/Users/terren/Documents/GitHub/web-server/plugin.js:92:34)
    at Object.getInstalledPlugins (/Users/terren/Documents/GitHub/web-server/plugin.js:245:30)
    at Timeout._onTimeout (/Users/terren/Documents/GitHub/web-server/index.js:210:102)
    at listOnTimeout (node:internal/timers:559:17)
    at process.processTimers (node:internal/timers:502:7) {
  errno: -2,
  syscall: 'stat',
  code: 'ENOENT',
  path: '/Users/terren/Library/Application Support/Simple Web Server/plugins/my_example/plugin.json'
}
could not import plugin my_example TypeError: fs.getByPath(...).text is not a function
    at getPluginInfo (/Users/terren/Documents/GitHub/web-server/plugin.js:92:60)
    at Object.getInstalledPlugins (/Users/terren/Documents/GitHub/web-server/plugin.js:245:30)
    at Timeout._onTimeout (/Users/terren/Documents/GitHub/web-server/index.js:210:102)
    at listOnTimeout (node:internal/timers:559:17)
    at process.processTimers (node:internal/timers:502:7)

No error in the UI. Here's the ZIP file I'm using. Would be great if you could try it: myplugin 2.zip

I'll work on fixing the bugs you identified. Thanks for pointing them out!

[terreng]

I fixed the file picker issue. As for fs.watch being unsupported, the best I can do is catch the error. It's totally not an essential feature, but unfortunately isn't available on Linux.

[ethanaobrien]

As for fs.watch being unsupported, the best I can do is catch the error. It's totally not an essential feature, but unfortunately isn't available on Linux.

Yeah, It is a little disappointing.

I also fixed support for zips zipped in a way the app doesn't like. It will scan the entire zip for the plugin.json file, and copy whatever folder it is in

Let me know if you find anything else

[terreng]

Great, thanks! Could you also fix the .DS_Store issue? Just ignore anything in the plugins directory that isn't a folder.

[ethanaobrien]

I just pushed my changes. Everything should be good

[terreng]

Great! Unless you find anything else, I think the release is ready to go.

The only thing left is documentation. I wrote a getting started page and documented the plugin.json format. Please write some documentation for plugin scripts, and then I can edit it after you're done if you'd like. Also remember to update/add documentation to the proxy plugin repo.

Once you do that, we can release this version!

[ethanaobrien]

@terreng

I'll get started on the documentation when I can

The proxy plugin itself is done (not yet the documentation) here is a compressed zip loadable as a plugin.

web-proxy.zip

After looking at this, I do urge you to reconsider my proposed idea on an option to disable some options in plugins that basically overwrite the entire handler, as it may cause confusion why a plugin like a proxy is going to do with directory or why most options do absolutely nothing. This might confuse the user, from my perspective, and I urge you to look at the plugin and think about it

[ethanaobrien]

@terreng Have we tried something like this

chokidar looks like they support it on linux, and they claim to do it more efficently

[terreng]

Awesome, looks like that handle the nightmare of having a bunch of recursive watchers. Good find, I'll update the code to use that and it should work on all platforms.

[ethanaobrien]

Also, looking at the comments of that stack overflow post, make sure to set awaitWriteFinish to true to make sure that if a file is being copied or something, that we won't get spammed with change callbacks

[ethanaobrien]

Hey @terreng I just wrote a more simplistic plugin that I was wondering if you wanted to add to the list of plugins.

Heres the repo url: https://github.com/ethanaobrien/cloudflare-ip-whitelist

It basically acts as a whitelist for cloudflare ip addresses. Cloudflare has the proxy feature that proxys all the requests, and this plugin blocks any request not from cloudflare ips

[terreng]

@ethanaobrien Sure, I added it to the list.

You code is fine, but there are a few improvements you could make:

• Load the ipv4 and ipv6 IPs in parallel instead of one then the other
• Think about what should happen in the error case
• Think about what should happen for requests that hit before the IP ranges have loaded (I'd suggest that you enqueue these requests and have them wait)

By the way, do custom dependencies work in plugins?

[ethanaobrien]

@terreng

Load the ipv4 and ipv6 IPs in parallel instead of one then the other

I thought about this, the main issue was in the case of an error. I didn't want to delete the already cached ips, so I just loaded one after the other, since it would be the easiest way to see if they both succeeded.

Think about what should happen in the error case

In case of an error, which is the same reason I didnt write it in parallel, it just keeps the old ips it already has.

Think about what should happen for requests that hit before the IP ranges have loaded (I'd suggest that you enqueue these requests and have them wait)

Or itd probably be a good idea to just have what's on cloudflare's site in the program, since the changelog shows it doesn't get updated too often, just to do this as a backup.

By the way, do custom dependencies work in plugins?

Yeah they do. They work in server side scripts too.