Modify NiFi Features docs page to primarily refer to inline docs

rjrudin commented 2 years ago

The docs page on NiFi Features - https://marklogic.github.io/nifi/nifi-features - is almost entirely duplicating the inline docs available with NiFi for each component of the connector. It is expected that the docs are not in sync with the code, which means that what the user reads in the docs may not match what's shown in NiFi.

The inline docs of controller services and processors, powered by code, is a real strength of NiFi. It allows a user to access docs at the exact point of when they use a feature.

The NiFi Features page can still have value for a new user in providing an overview of the components that are available. That could simply be a list of each controller service and processor and then a copy/paste of the description of each. Properties/relationships would all be better described in the inline docs.

To make this change, we'd need to ensure that we don't lose any information that's in the docs that is not yet in the inline docs. But we'd likely want to do that regardless, since a user seems far more likely to go with what's in the inline docs vs at our docs website.

rjrudin commented 2 years ago

@dmcassel @17llamas Let me know what you think about this change. I am expecting that most users make heavy use of the inline docs in NiFi and rarely, if ever, go to the docs website mentioned above.

DavidEnnis-CleverLlamas commented 2 years ago

@dmcassel @17llamas Let me know what you think about this change. I am expecting that most users make heavy use of the inline docs in NiFi and rarely, if ever, go to the docs website mentioned above.

Personally, I prefer a single book of record for official docs. Supporting information elsewhere can be great (but clearly supporting information and not meant to be API docs).

MarkLogic NiFi is a good example:

Some ML inline docs driving the NiFi interface are thin.
But then some documentation on the website is out of date or simply a mirror of the inline docs.

For NiFi, I agree - make clear and robust documentation in the controllers(driven by inline docs) that can be used without necessarily needing external references. However, if you are to do this, I think it's important to further refine the inline docs.

An example: ExecuteScriptMarkLogic

has a property to skip the first result
but also has a relationship "first result" -- for me, this is all confusing. ---- First off, why give an option for skip result if I am executing arbitrary code..? I can just suppress the first record in my code, if need be. -- But then: If I skip the first result. I still have a result-set

In this case, how the use of "skip first result" property affects the "first result" relationship can be clearly determined by the generated docs for the property. It does not end up in either relationship (fully skipped). However, how the relationships work when not skipping is not clear nor is it clear if "results" is inclusive of first and last.

Looking at the relationships, if I do not skip first result, and I have a set of 8 records, does my relationship usage end up as first: 1 - results: 6 - last: 1 = 8 (this scares me - do I miss 2/6 records if I do not route first and last to my destination?) OR first: 1 -results: 8 - last: 1 = full set to results and copy to first/last

And what if I have one record? do I get 1,1,1 or 0,1,0 (I am first, record and last)

This is where I would expect online docs to perhaps go further with samples. - but it does not, so an example of no added value there. In this case, it is the relationship inline documentation that could do with clarity about how they work together. Instead, it's off to the code (I have gotten used to just leaving the codebase open and answering the remaining questions as I go.)

With more clear inline docs (And drop the duplication online), I should not have to read this and build a little truth table in my head to infer the answer:


for (EvalResult result : call.eval()) {
                count++;
                // get result as string
                String resultStr = result.getString();
                last = resultStr;

                if (count == 1) {
                    FlowFile firstFF = session.create(originalFF);
                    resultToFlowFile(session, resultStr, firstFF, resultsDest);
                    session.transfer(firstFF, FIRST_RESULT);
                }

                if (count > 1 || !skipFirst) {
                    FlowFile resultFF = session.create(originalFF);
                    resultToFlowFile(session, resultStr, resultFF, resultsDest);
                    session.transfer(resultFF, RESULTS);
                }
            }

            originalFF = session.putAttribute(originalFF, MARKLOGIC_RESULTS_COUNT, Integer.toString(count));
            session.transfer(originalFF, ORIGINAL);

            if (last != null && (count > 1 || !skipFirst)) {
                FlowFile lastFF = session.create(originalFF);
                resultToFlowFile(session, last, lastFF, resultsDest);
                session.transfer(lastFF, LAST_RESULT);
            }`

rjrudin commented 1 year ago

The "NiFi Features" page will be updated in 1.16.3.2 to briefly describe the components and refer the user to the "View Usage" feature for each processor, which provides visibility to everything that's been documented via code.

For the comment above - providing better examples will be a separate effort.

DavidEnnis-CleverLlamas commented 1 year ago

The "NiFi Features" page will be updated in 1.16.3.2 to briefly describe the components and refer the user to the "View Usage" feature for each processor, which provides visibility to everything that's been documented via code.

For the comment above - providing better examples will be a separate effort.

Great news! Thanks

marklogic / nifi

Modify NiFi Features docs page to primarily refer to inline docs #153