search

bbende / nifi

Mirror of Apache NiFi
Apache License 2.0
1 stars 1 forks source link

Implement a 'quick-import' command #41

Closed aperepel closed 6 years ago

aperepel commented 6 years ago

Capturing some ideas here to have a place to discuss them. Here's what I envision.

As a user, I want to copy a public URL of the flow json (e.g. Gist) and have it up and running in my NiFi instance with next to no additional steps

Assumptions:

  1. Both NiFi and NiFi Registry are installed and NiFi configured to use the registry.
  2. The bucket and flow names/versions are pre-defined and not configurable by a user. I.e. there's a QuickImport bucket, and the flow object will have the QuickImport- name + some generated description. I.e. it's a one-time operation and we don't expect to be automatically add versions to that flow.
  3. PG is imported into a NiFi canvas at a root level, using the default layout code already in the CLI. No custom placement or adding to a nested group.
  4. PG is not started automatically.
  5. A user is encouraged to move the versioned flow to a different bucket (by stopping the VC support and enabling again for now). A 'move to a different bucket' macro action could be implemented later.
  6. The action uses the 1 registry client automatically. Support for specifying an rcid (when multiple are configured) can be added later.
  7. The QuickImport bucket is just a convention, but we may consider giving it some special protected meaning in the NiFi registry in the future.
aperepel commented 6 years ago

Ok, so I took a stab at implementing this, but quickly came to a conclusion we would need some refactoring to have more reusable actions and decouple the action and output printer more. E.g. I ended up copying a lot of action code to implement a logical sequence like:

bbende commented 6 years ago

@aperepel I want to clarify a couple of things to make sure I understand the functionality...

So user runs a command like: quick-import -i http://foo.com/flow.json

Behind the scenes this:

  1. Creates a bucket named QuickImport
  2. Creates a flow in the QuickImport bucket, named QuickImport Flow
  3. Imports the JSON from the URL as v1 of QuickImport Flow
  4. Runs pg-import against NiFi to import v1 of QuickImport Flow

Does that seem correct? Just want to make sure I get the right idea while trying to think about how to support this.

aperepel commented 6 years ago

Yes, here are the assumptions:

  1. QuickImport bucket is created on demand
  2. The flow name is QuickImport-<millistimestamp> (so it can be sorted easily). Description like Auto-import on (full date) from (source URL).
  3. There will be only 1 version per flow. Every run will create a new timestamped flow.
  4. Pg import. Ultimately a user only needs to refresh nifi canvas or wait the default 30 secs.
bbende commented 6 years ago

Got a version of this working in master, quite fun to play with :)

For now I introduced a new group called "demo" where we can put other macro commands, but if we don't like that we can either rename that group, or move the command to top-level under no group.

demo quick-import -i https://gist.githubusercontent.com/aperepel/3653bd359740519efe373a65d0d94db4/raw/af8af15d1ac375be8bedbe9e2f66adef471bb678/test_flow.json