More powerful plugin API proposal

[karaggeorge]

Plugins API

Looking for feedback on couple of thoughts around ways to improve our plugin API

Reasoning

There's a lot of requested features for Kap and I know we try to not add new settings/preferences, but it gets hard to manage all of that. I think a good solution is for these features to be plugins. The current plugin API is a bit limiting (only exporting). We should try to expand it a bit more and allow for a bigger variety of plugins, that can take the load off of Kap. This will also help people only have the functionality they want.

I've been thinking about a few of these for a while, but I wanted to put it in writing, so I can keep track and get other people's thoughts on the matter before I start implementing parts

APIs

Exports

This is the only type of plugin we support at the moment. I think the API we currently have is good, but we might experiment with more options, like:

• Allowing plugins to add/modify ffmpeg arguments (use cases: slow motion, speed up)
• Allow plugins to not use the default ffmpeg process we provide but handle the export completely on their own (more complex ffmpeg conversions)
• Allow plugins to provide error messages (we let them provide loading text, should be able to have them show a meaningful error when the export fails)
• Maybe somehow allow export plugins to be connected? As in, Recording -> Reverse plugin -> Slow mo plugin -> Export to GIF
  • For this it might be worth having two types of plugins, one being editing and one being export (editing being mp4 -> mp4, exporting being mp4 -> target)

Recording

The way I'm picturing this is with some hooks (maybe an event emitter) that allows plugins to run code before the recording starts, and after it's done (maybe separate for error/success). This would allow current built-in features like Hide Desktop Icons/DND/Key Cast/Custom Cursor to be plugins.

Deep Linking

For any type of plugin, I think deep linking is something that might be useful for OAuth with external services. We could have the format of the default url be something like kap://plugin/{pluginName}/some/path and the app is responsible of taking that and sending /some/path to the right plugin.

Implementation-wise, I'm not sure what the best way to do it. OAuth is generally an async process, but the plugin will be probably opening a link to the login page and waiting for a response. So maybe a waitForDeepLink() function that just waits for that deeplink and resolves when it receives it? Or maybe have them define a general deep link callback, which then can take care of writing whatever data it needs (token) to the store async from when the login happened? (either way we can document it)

Settings

It might be worth letting plugins have their own settings UI, if it's more complicated than plain JSON Schema. Since they can have electron as a peer-dependency, they can create their own window and render their own preferences. Use-case for this is for example, for a possible custom cursor plugin, a UI to let the user select between cursors.

Metadata

• We added a field that plugins can use to define what version of Kap they are compatible with. We might want something similar with what version of Kap Plugin API they are compatible with, unless we keep it 100% backwards compatible with the current API.
• Having a plugin type (editing/export/recording) might be useful to filter/separate them somehow in the preferences window

Changes to current API

I don't think I've seen any plugin export more than one of what we call services. It sounds like it would allow one npm module add two different plugins (maybe related), but I haven't seen a use-case for it yet. I think it might simplify both the developing of plugins and the code for reading them if we assumed each module only exports one plugin, with one set of settings.

Other Thoughts

I'm still trying to figure out how linking multiple "editing" plugins could work UI-wise, or if that's something we consider out of scope for Kap, even when it's implemented through plugins, since it's a screen recorder, not really an editor. We provide some editing (size/fps/cropping), that are all supported through ffmpeg commands. So maybe supporting plugins adding/modifying ffmpeg arguments is enough?

[gotjoshua]

Wow, 21 days and no comments yet : ( @karaggeorge, @dreamorosi @ripper2hl and other fork owners, any feedback? )

each module only exports one plugin, with one set of settings. +1

So maybe supporting plugins adding/modifying ffmpeg arguments is enough? +1

I really like the idea of plugin types and the metadata proposal above.

I wonder about 2 other types of plugins that were not mentioned: 1) theme - for visual UX changes 2) a plugin that can influence core settings/functionality - like replace the whole bottom bar on the video editor (not just add to it)... this is not fully thought through, but ( as mentioned here ) i'd really love to have more flexible framerate options. If those don't make it into the core UI ( which i am still really hoping for), then I'd love to have a way to influence the core UI without maintaining a fork.

[ripper2hl]

Sorry but I only create a fork for the incompatibility with Linux and works fine in Ubuntu but need more work.

The proposal is very nice, if considering GNU/Linux.

[sindresorhus]

@ripper2hl We don't have any intention of supporting Linux.

[sindresorhus]

This looks great as a high-level proposal. I do think we should split it up into individual issues to make it easier to have focused discussions about the different features though.

Allowing plugins to add/modify ffmpeg arguments (use cases: slow motion, speed up)

This should be simple to implement and could be very powerful.

Allow plugins to not use the default ffmpeg process we provide but handle the export completely on their own (more complex ffmpeg conversions)

Example use-cases?

Allow plugins to provide error messages (we let them provide loading text, should be able to have them show a meaningful error when the export fails)

I consider this a bug, not an enhancement. I don't see how we need an API for this. We should just present the error the plugin throws. (https://github.com/sindresorhus/kap-gifski/issues/3#issuecomment-582263116)

For this it might be worth having two types of plugins, one being editing and one being export (editing being mp4 -> mp4, exporting being mp4 -> target)

:+1: This is discussed in https://github.com/wulkano/kap/issues/550.

Recording The way I'm picturing this is with some hooks (maybe an event emitter) that allows plugins to run code before the recording starts, and after it's done (maybe separate for error/success). This would allow current built-in features like Hide Desktop Icons/DND/Key Cast/Custom Cursor to be plugins.

:+1: This could enable a lot of use-cases and should not be difficult to implement.

Deep Linking So maybe a waitForDeepLink() function that just waits for that deeplink and resolves when it receives it?

This sounds like the best way to do it. await waitForDeepLink(), and it returns everything after {pluginName} in kap://plugin/{pluginName}/some/path.

Settings Use-case for this is for example, for a possible custom cursor plugin, a UI to let the user select between cursors.

Couldn't we just let the plugin add their own JSX to the generated settings window?

Metadata We added a field that plugins can use to define what version of Kap they are compatible with. We might want something similar with what version of Kap Plugin API they are compatible with, unless we keep it 100% backwards compatible with the current API.

How is the Kap plugin API version different from the Kap version? I don't see why I need to specify plugin API version. The API is updated with Kap. It should be enough to just specify the Kap version.

Having a plugin type (editing/export/recording) might be useful to filter/separate them somehow in the preferences window

We could have icons to indicate what the plugin provides. But do we need metadata for that? We already load the plugin, so we should just check .shareServices.length?

I don't think I've seen any plugin export more than one of what we call services. It sounds like it would allow one npm module add two different plugins (maybe related), but I haven't seen a use-case for it yet.

I modeled that after the macOS share services API. While it hasn't been used yet, there aren't that many plugins either. Probably because we haven't marketed it strongly. There's not even a link on the website to the plugin docs. So I wouldn't take that as much proof yet. How will it simplify developing plugins to only allow one share service? I assume a plugin package could have both a "share service" and an "editing service"? So there's still going to be multiple things in a package, and I think that's fine.

I'm still trying to figure out how linking multiple "editing" plugins could work UI-wise, or if that's something we consider out of scope for Kap, even when it's implemented through plugins, since it's a screen recorder, not really an editor. We provide some editing (size/fps/cropping)

I don't know either, but I don't think it's a high-priority, so we can look into it later on when other things are implemented.

[karaggeorge]

@sindresorhus Thanks for the feedback. I can start implementing a couple of the smaller stuff and splitting some in their own issues 👍

Some comments/questions:

Allow plugins to not use the default ffmpeg process we provide but handle the export completely on their own (more complex ffmpeg conversions)

Example use-cases?

For example a gif conversion has 2 ffmpeg calls, so I'm not sure how we would allow for ffmpeg arguments to be edited with the first idea. Also for example when I was trying to get the alternate cursor with the original method (generating a video of just the cursor and overlapping) that also needed two ffmpeg calls, before the actual final conversion one (one to generate the transparent video and one to overlay them).

Settings Use-case for this is for example, for a possible custom cursor plugin, a UI to let the user select between cursors.

Couldn't we just let the plugin add their own JSX to the generated settings window?

Would they export a react component? And would that overwrite our default UI, or would it extend it (include it at the top/bottom)

Having a plugin type (editing/export/recording) might be useful to filter/separate them somehow in the preferences window

We could have icons to indicate what the plugin provides. But do we need metadata for that? We already load the plugin, so we should just check .shareServices.length?

But how could we tell the type? I guess an export one would have the export action, recording would have hooks, and editing would have ffmpeg calls? As in, are we determining the type from the type of keys in the service?

[ripper2hl]

@ripper2hl We don't have any intention of supporting Linux.

Okay, i understand, not problem if the proposal help to simplify the code this simplify my fork :smile:

Good ideas but need more work in Aperture for add some FFmpeg filters and i dont know what filter try to add D:

[sindresorhus]

For example a gif conversion has 2 ffmpeg calls, so I'm not sure how we would allow for ffmpeg arguments to be edited with the first idea. Also for example when I was trying to get the alternate cursor with the original method (generating a video of just the cursor and overlapping) that also needed two ffmpeg calls, before the actual final conversion one (one to generate the transparent video and one to overlay them).

Maybe we just give them access to what arguments we use and also the ffmpeg instance and let them do their thing however they like. Then they can choose to use the arguments we use for each call, tweak them, or use their own.

Would they export a react component? And would that overwrite our default UI, or would it extend it (include it at the top/bottom)

Export React component, yes. It would extend. Probably top, but could also be configurable, I guess.

But how could we tell the type? I guess an export one would have the export action, recording would have hooks, and editing would have ffmpeg calls? As in, are we determining the type from the type of keys in the service?

I assume each service would have a separate export:

exports.shareServices = […];
exports.editingServices = […];
exports.recordingServices = […];

So we just check the length of those arrays.

[karaggeorge]

Maybe we just give them access to what arguments we use and also the ffmpeg instance and let them do their thing however they like. Then they can choose to use the arguments we use for each call, tweak them, or use their own.

Might be a bit tricky still for the gif conversions, since it's two of them, which set of arguments to we provide?

Other way I can think of it (since gif is the only one that breaks the norm) we run the first ffmpeg command to generate the palette, and then just provide the palette path as part of the ffmpeg args for the second one. (pretty much treating the gif conversion as one ffmpeg call, the second one, which just requires some async preprocessing)

[sindresorhus]

Other way I can think of it (since gif is the only one that breaks the norm) we run the first ffmpeg command to generate the palette, and then just provide the palette path as part of the ffmpeg args for the second one. (pretty much treating the gif conversion as one ffmpeg call, the second one, which just requires some async preprocessing)

That's a good idea! 👌

[karaggeorge]

@sindresorhus I think all of the comments here were implemented in those last few PRs. I think we can close this and open new issues for anything else that comes up?