feat: Add CRUD commands for core resources

alexluong commented 4 months ago

This PR adds several CRUD-y commands for Source, Destination, Connection resources.

Demo: https://share.cleanshot.com/6NHV1l06

Technical Callouts

1: I'm using huh for the form library. We're currently using survey which is no longer maintained. The author recommends huh as an alternative. I also consider the other suite of libraries from the folks at Charm and think they could be a good fit for us.

2: I refactored the Cobra command setup a little bit. The main motivation is that when we add or remove new commands, we only need to deal with that command file directly without having to modify another file (cmd/root.go) like we have to right now.

Within this refactor, I introduced 2 different patterns that you can observe for hookdeck connection commands vs hookdeck source commands. The difference is connection splits each command into its own file while all source commands are in the same file. The main difference is the number of files vs the length and complexity of an individual file. I don't have a strong preference here and believe either can work just fine. Curious to hear if you have any strong preference here.

3: I introduced 2 new internal packages within the codebase, forms and tui.

forms is responsible for rendering forms & constructing payloads to make API calls. tui is responsible for rendering the UI of our resources. Currently, it's very straightforward but I can imagine it being much more complex in the future if we want to employ tables and custom layouts.

Currently, the cmd are using forms and tui, and everything seems to work okay for now. If things become a bit more complex, we may consider adding a controllers package to help with more custom logic.

This organization is a bit different from our existing approach where we have a package for each command group like listen, login, logout, etc. That approach will work fine but as we continue expanding our CLI, I'm concerned that things may get a bit more messy than we'd like.

Curious to hear your thoughts on this topic as well!

Next steps

[ ] Improve creation forms. Currently, source & destination forms only have a name option, which is not super helpful yet, especially for destinations. We should consider adding more customization here to get to a maximum usefulness while not being too complex (connection rules, source verification, for example)
[ ] Improve UI. Currently, a resource UI only shows its ID and its name, which is good enough for identification while development but is not super useful when it comes to real production usage. We should consider adding more information so it's more useful for developers using the CLI in their development workflow.
[ ] Add update form
[ ] Add other functionality like disable / enable
[ ] Improve listing / querying logic (ordering and pagination)

Concerns

1: In the connection creation form, we allow the user to select an existing resource (source / destination) or create a new one. The concern is what if a user has a lot of resources (say 1000 sources). We have to figure out a way to integrate pagination or search logic into the form or some way for users to specify an existing ID manually. This seems like a non-trivial problem that may require some UX design for the terminal.

2: In the connection creation form, when moving to a new section, we're not showing the existing form state. For example, when we're in the connection section, we don't see which source and destination data that was entered. We should improve the UX here as well.

leggetter commented 4 months ago

@alexluong - thanks. I think this is a useful experiment to begin a conversation.

@alexbouchardd feels that we should take a more config-first approach and simplify the usage of the CLI to something like the Terraform Provider where there's just a hookdeck apply --config-file path/to/hookdeck.config.[json|yml].

CLI update goals

One of the goals that I have of adding connection CRUD to the CLI is to improve onboarding and get a developer to the point of seeing Hookdeck in action with minimal effort and all from their own environment.

The more steps a developer has to perform, the longer it takes them to get to an Aha! moment. As above, interactivity in the CLI should be a fallback, so we should support params/flags that enable the quickstart functionality to be performed without the need for interactive prompts.

For example, in the receive webhooks quickstart:

hookdeck connections create --name inbound-quickstart-connection \
--source-name inbound-webhook \
--destination-name outbound-webhook \
--destination-url https://mock.hookdeck.com

Note: I've gone with named flags over parameters. But we should have a set of required parameters instead.

Passing params/flags vs. interactive

So the Forms stuff is for interactive functionality?

If so, the pattern we should follow is:

Support params / flags being passed directly to commands
If required params/flags are not passed, then fallback to interactive prompts

Am I correct in saying we haven't done 1. here?

Use singular noun commands for actions and plural for resources

Our SDKs use the singular terminology e.g. connection vs. connections. This is wrong, but it's a breaking change in the SDKs. IMO we should be using plural.

@alexbouchardd do you disagree? This branch uses a singular workspace. https://github.com/hookdeck/hookdeck-cli/pull/66 is a breaking change but also opts for the singular of project rather than projects. So, maybe we prefer the singular?

alexbouchardd commented 4 months ago

@leggetter I have no strong opinion on singular vs plural, I think we should benchmark and follow the most popular approach.

The more steps a developer has to perform, the longer it takes them to get to an Aha! moment. As above, interactivity in the CLI should be a fallback, so we should support params/flags that enable the quickstart functionality to be performed without the need for interactive prompts.

Realistically, the only use for flags is copy-pasting from a guide. They're too verbose and not used commonly enough to expect any user to do this. That might still be a valid use case, but personally, I think that doesn't set up the use for success. It gets you from 0 to 0.5 quickly but does it through a method that you are unlikely to use for the real use case. I'd argue it makes the 0.5 to 1 harder. That's my reservation about it.

The upside with something like hookdeck apply --config-file path/to/hookdeck.yml is that it's something you can build on easily and a pattern you might actually use for real.

I'd argue the form-based approach is also more likely to get used, BUT at that point, it's not clear to me how a form in your terminal would be better than one in the dashboard.

alexluong commented 4 months ago

@alexbouchardd feels that we should take a more config-first approach and simplify the usage of the CLI to something like the Terraform Provider where there's just a hookdeck apply --config-file path/to/hookdeck.config.[json|yml].

Interesting. So the idea is we'll have a "Hookdeckaform" of some sort that kinda works like Terraform? My first reaction to that idea is whether we need to implement a state storage of some sort to keep track of the previous state before applying the new config. The tricky case is how to delete resources?

One other concern is how should resources in the configuration handle id? Or would we need to match things by name? What about connection in that case?

Regardless, let me noodle on this idea a bit more. For the time being, I think we should consider leaning on established conventions such as Terraform for this purpose, but I do think we can do something here to support non-Terraform users better.

Am I correct in saying we haven't done 1. here? (1 being "Support params / flags being passed directly to commands")

Yes, this is correct. I did consider this briefly but I agree with @alexbouchardd that this will add a lot of scope to the command while not providing a lot of value. Because the resources can get complicated fast, especially connections, I'm concerned that named params may not be enough.

The biggest downside of our form-based approach is it's not friendly to scripting which is an important aspect of a CLI tool.

I do think we should have a solution if folks want to do more scripting with Hookdeck CLI. I just think it's a tad beyond the scope of our current experiment.

it's not clear to me how a form in your terminal would be better than one in the dashboard.

My main point on this is that the form in the terminal will meet developers where they're already at during the development process so they can do everything from the command line without switching back between tools.

With that said, I'm thinking more than just a form but rather recreating everything a developer can do on the dashboard, from viewing events / requests to enabling / disabling resources and even managing bulk retries some day, etc. That way, the CLI can be another avenue for users to interact with Hookdeck. I think this also helps show that Hookdeck is a serious developer tool and infrastructure company when developers can manage everything from their terminals. A comparison would be AWS CLI vs AWS dashboard here.

alexbouchardd commented 4 months ago

With that said, I'm thinking more than just a form but rather recreating everything a developer can do on the dashboard, from viewing events / requests to enabling / disabling resources and even managing bulk retries some day, etc.

That's a valid idea, but I overall feel skeptical that's worth the effort right now. It would come with a huge maintenance and overhead burden that we probably do not want to have at this time?

Interesting. So the idea is we'll have a "Hookdeckaform" of some sort that kinda works like Terraform? My first reaction to that idea is whether we need to implement a state storage of some sort to keep track of the previous state before applying the new config. The tricky case is how to delete resources?

Agreed it's tricky and does feel like reinventing the wheel a bit. All our resources have idempotent endpoints so state is not that complicated (except for delete, but we could just not support delete...)

The biggest downside of our form-based approach is it's not friendly to scripting which is an important aspect of a CLI tool.

Definitely, but can we think of a huge case where scripting would be needed? Wouldn't most people just want to use the API / SDK in that case?

alexluong commented 4 months ago

@alexbouchardd all fair points here. Any lean on which direction you'd like to explore further?

Maybe a good way to evaluate is to answer the "why", as @leggetter mentioned, and see which path makes the most sense?

leggetter commented 4 months ago

For full public visibility: We won't progress with the commands for now. There's an agreement that there's something in between API usage and the Hookdeck Terraform Provider that's missing. But we're not sure what that is yet.

hookdeck / hookdeck-cli