search

uniberg / kbn_sankey_vis

Sankey diagram for Kibana visualize.
Apache License 2.0
28 stars 11 forks source link

visualization doesn't work in Kibana 7.9.x #15

Closed mmguero closed 2 years ago

mmguero commented 3 years ago

I don't know if you're looking forward at all to the Kibana 7.9.x and beyond, but the kbn_sankey_vis plugin won't work as relies on stuff in the legacy plugin format that was removed.

I'm really not a very good Kibana plugin developer yet, but I have done a little bit of research trying to figure out why. I'll refer you to this thread:

Debugging missing plugins after updating from 7.6.2 to 7.9.2

The advice I got there was to either put in a shim/compatibility to temporarily fix it for 7.9.2, or to migrate to the "new plugin framework."

I'm not really knowledgeable enough at this point to do either, but I wanted to log it here to see if it was on your radar and at least have somewhere we could track it.

Thanks.

ch-bas commented 3 years ago

Hi @mmguero, Thanks for your suggestion and interest in our plug-in. We are currently thinking about a total code refactoring for the whole plug-in in order to work with the new Kibana platform. We will keep you updated about the new outcome soon.

mmguero commented 3 years ago

Very excited about this, thanks for the update.

bfgiordano commented 3 years ago

That would be great to get the plugin available moving forward. Appreciate it!

On Mon, Nov 2, 2020 at 3:35 PM SG notifications@github.com wrote:

Very excited about this, thanks for the update.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/uniberg/kbn_sankey_vis/issues/15#issuecomment-720709567, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIYKVPSBUNPGSLNDNAFKNDSN4JYPANCNFSM4TH32T6A .

hakanyildiz commented 3 years ago

Hi,

Thx for great plugin. However, It does not work in Kibana 7.10.* . Is there will be any update?

ch-bas commented 3 years ago

Hi @hakanyildiz, so far the update for the new Kibana platform is planed for next year. We will keep you updated for the new releases. Thanks!

ch-bas commented 3 years ago

Hi @mmguero @bfgiordano @hakanyildiz , a new pre-release was published today, it is still under testing but a final version should be ready in the upcoming days.

bfgiordano commented 3 years ago

Awesome - thank you for continuing to move this plugin forward. Barbara

Sent from my iPhone

On Feb 19, 2021, at 11:00 AM, bassem chagra notifications@github.com wrote:

 Hi @mmguero @bfgiordano @hakanyildiz , a new pre-release was published today, it is still under testing but a final version should be ready in the upcoming days.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

mmguero commented 3 years ago

Thank you! You really made my day with this. In my opinion this is one of the essential third party visualizations, and I've looked at a lot of them. I will test out this prerelease today.

mmguero commented 3 years ago

I tried downloading, building and installing the plugin as of commit 8ac7a8bffc018716b94747ecb328c771ff42b63a this morning, and ran into the following issue:

The plugin installed okay, but when loading Kibana I see this in the browser:

Elastic did not load properly. Check the server output for more information.

Looking at my NGINX logs:

nginx-proxy_1    | 172.22.26.1 - user [19/Feb/2021:21:03:01 +0000] "GET /kibana/app/visualize HTTP/1.1" 200 17244 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0"
nginx-proxy_1    | 172.22.26.1 - user [19/Feb/2021:21:03:01 +0000] "GET /kibana/bootstrap.js HTTP/1.1" 200 2354 "https://localhost/kibana/app/visualize" "Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0"
nginx-proxy_1    | 172.22.26.1 - user [19/Feb/2021:21:03:01 +0000] "GET /kibana/36136/bundles/plugin/kbnSankeyVis/kbnSankeyVis.plugin.js HTTP/1.1" 404 60 "https://localhost/kibana/app/visualize" "Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0"

I don't see a kbnSankeyVis.plugin.js anywhere in my Kibana container, and I'm not sure what the 36136 thing would be about either. Perhaps the installations for building and installing a release .zip are different in the new version of Kibana? My build process looks like this, which more or less lines up with the instructions at the bottom of the README:

curl -sSL -o /tmp/kibana-sankey.zip "https://codeload.github.com/uniberg/kbn_sankey_vis/zip/feature/update_7.10.1" && \
cd /tmp && \
  echo "Installing Sankey visualization..." && \
  unzip /tmp/kibana-sankey.zip && \
  mkdir ./kibana &&\
  mv ./kbn_sankey_vis-* ./kibana/sankey_vis && \
  cd ./kibana/sankey_vis && \
  npm install --production && \
  cd /tmp && \
  zip -r sankey_vis.zip kibana --exclude ./kibana/sankey_vis/.git\* && \
  cd /usr/share/kibana/plugins && \
  /usr/share/kibana/bin/kibana-plugin install file:///tmp/sankey_vis.zip --allow-root && \
  rm -rf /tmp/kibana /tmp/*sankey*

I don't seem to get errors during the install of the plugin:

Installing Sankey visualization...
Archive:  /tmp/kibana-sankey.zip
8ac7a8bffc018716b94747ecb328c771ff42b63a
   creating: kbn_sankey_vis-feature-update_7.10.1/
 extracting: kbn_sankey_vis-feature-update_7.10.1/.eslintignore  
  inflating: kbn_sankey_vis-feature-update_7.10.1/.eslintrc.yml  
...
  inflating: kbn_sankey_vis-feature-update_7.10.1/test/filterTest.js  
kbn-sankey-vis@7.10.2 /tmp/kibana/sankey_vis
+-- d3-plugins-sankey@1.2.1  (git+https://github.com/uniberg/d3-plugins-sankey.git#c1ec50f3d3e88ec5cc5b2a9ab8e7987ce38ea6cd)
| `-- d3@3.5.17 
`-- json-stable-stringify@1.0.1 
  `-- jsonify@0.0.0 

  adding: kibana/ (stored 0%)
  adding: kibana/sankey_vis/ (stored 0%)
...
  adding: kibana/sankey_vis/node_modules/json-stable-stringify/test/to-json.js (deflated 63%)
Attempting to transfer from file:///tmp/sankey_vis.zip
Transferring 677223 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete

If this isn't helpful, or not applicable while you're still in pre-release, then please ignore this comment. I'm fine waiting until your done. Just wanted to help as much as possible.

ch-bas commented 3 years ago

Hi @mmguero , thanks for reporting that! I updated the readme file to include the new instructions for building the plugin. Could you follow them and see if your problems will be solved? Thanks!

mmguero commented 3 years ago

Here's what I'm running into with the updated build release instructions:

[root@000bdbb111ba tmp]# git clone --single-branch --recursive --depth=1 --branch=feature/update_7.10.1 https://github.com/uniberg/kbn_sankey_vis.git sankey_vis
Cloning into 'sankey_vis'...
remote: Enumerating objects: 42, done.
remote: Counting objects: 100% (42/42), done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 42 (delta 4), reused 29 (delta 4), pack-reused 0
Unpacking objects: 100% (42/42), done.
[root@000bdbb111ba tmp]# cd sankey_vis/
[root@000bdbb111ba sankey_vis]# yarn install
yarn install v1.22.5
info No lockfile found.
[1/4] Resolving packages...
warning mocha > mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 9.15s.
[root@000bdbb111ba sankey_vis]# yarn build --kibana-version 7.10.2
yarn run v1.22.5
$ yarn plugin-helpers build --kibana-version 7.10.2
$ node ../../scripts/plugin_helpers build --kibana-version 7.10.2
internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module '/scripts/plugin_helpers'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[root@000bdbb111ba sankey_vis]# node --version
v10.23.3
[root@000bdbb111ba sankey_vis]# npm --version
6.14.11
[root@000bdbb111ba sankey_vis]# yarn --version
1.22.5

The important parts of my build environment (in a Docker image) can be seen as such:

RUN yum install -y epel-release && \
      yum update -y && \
      curl --silent --location https://rpm.nodesource.com/setup_10.x | bash - && \
      curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | tee /etc/yum.repos.d/yarn.repo && \
      rpm --import https://dl.yarnpkg.com/rpm/pubkey.gpg
mmguero commented 3 years ago

I'm wondering if maybe it matters where I am in the filesystem (e.g., somewhere under the /usr/share/kibana source tree?) where I clone the repo?

mmguero commented 3 years ago

What I'm really wondering is if my Docker image is no longer sufficient to build the plugin from scratch. I'm going to research more in the Kibana contributing guide and see what I might be missing.

ch-bas commented 3 years ago

Running 'yarn build' will invoke using some dependencies from Kibana, so unless you have cloned the Kibana repository in the docker container and installed the dependencies(yarn kbn bootstrap), you will not be able to use the build process like you are doing right now. I suggest to use the docker image provided by the Elastic team in docker hub and simply install the plugin using the normal way( kibana-plugin install zip file).

mmguero commented 3 years ago

I understand now, I did some more reading about the plugin development environment for Kibana 7.10+ and have adjusted my build process. I'm getting it built now and will report back with the results on how the plugin works.

mmguero commented 3 years ago

I got it installed correctly and it's mostly working. I have noticed a few issues:

  1. If I run the visualization over a time frame with no data present I get a warning dialog with "Minimum two sub aggs needed."
  2. If I have run the visualization as part of a dashboard and correctly see data in the graph, then if I do something to invalidate the canvas (for example, resize my browser window) I get "No results found." Hard-refreshing the page will then cause it to render correctly.
mmguero commented 3 years ago

One more I found:

When trying to use the "Group other values in separate bucket" visualization option, I get this search error:

sankeyProvider/<@https://localhost/kibana/36136/bundles/plugin/kbnSankeyVis/kbnSankeyVis.plugin.js:26:218644
sankeyProvider@https://localhost/kibana/36136/bundles/plugin/kbnSankeyVis/kbnSankeyVis.plugin.js:26:218418
_callee$@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:215955
tryCatch@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:32004
invoke@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:35976
defineIteratorMethods/</prototype[method]@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:33129
visualization_function_asyncGeneratorStep@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:212591
_next@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:212965
ch-bas commented 3 years ago

Thanks for reporting! I will try to take a look soon.

ch-bas commented 3 years ago

@mmguero , the mentioned issues were fixed in the new pre-release.

mmguero commented 3 years ago

Thanks! I tested it and the issues mentioned in this comment https://github.com/uniberg/kbn_sankey_vis/issues/15#issuecomment-784441544 are fixed. I still get an error when I try "Group other values in separate bucket" or "Show missing values":

bucketHelper(...) is undefined

sankeyProvider/<@https://localhost/kibana/36136/bundles/plugin/kbnSankeyVis/kbnSankeyVis.plugin.js:26:224456
sankeyProvider@https://localhost/kibana/36136/bundles/plugin/kbnSankeyVis/kbnSankeyVis.plugin.js:26:224230
_callee$@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:215955
tryCatch@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:32004
invoke@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:35976
defineIteratorMethods/</prototype[method]@https://localhost/kibana/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:33129
visualization_function_asyncGeneratorStep@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:212591
_next@https://localhost/kibana/36136/bundles/plugin/visualizations/visualizations.plugin.js:6:212965
ch-bas commented 3 years ago

HI @mmguero , could you elaborate how to reproduce the bug, and in which Kibana's version is this happening ? I was testing with Kibana 7.10.2, with 4 layers defined in the Sankey visualisation but couldn't see that problem. Thanks in advance!

mmguero commented 3 years ago

Sure. I'm using Kibana 7.10.2 as well. Here's my entire build process for the plugin.

All I'm doing is setting up a sankey visualization with 3 layers, going into the visualization editor and choosing any of the buckets, and enabling "group other values in separate bucket" or "show missing values." I have attached a video file here, does this match what you are doing? And you're not getting that error?

https://user-images.githubusercontent.com/13872653/110492591-92797e80-80af-11eb-9889-832ca77ce6ce.mp4

ch-bas commented 3 years ago

@mmguero , the new pre-release should fix the issue you mentioned, thanks again for explaining how to reproduce it!

mmguero commented 3 years ago

Hmmm, I'm still getting the same error. Looking at your comment with the commit and the contents of the source, I'm not sure you fixed the same thing I was talking about (although there certainly may have been an issue with custom labels as well?).

The issue I'm seeing is not with "Custom label", but is with "Group other values in separate bucket" and "Show missing values."

"Group other values in separate bucket," if enabled, will do just that. Suppose you have "Size" set to 5. If "Group other values in separate bucket" is disabled you will only have 5 values displayed, even if you have 100 unique values. However, if you have "group other values in separate bucket" enabled, you'd have 6 values displayed: your top 5, then another bucket that says "Other" (or whatever you call it) that contains the sum total of the other 95 values.

Similarly, "Show missing values" will create an extra bucket for any documents that do not contain a value for the field chosen.

Even with your latest update, I am getting the bucketHelper error with either "Group other values in separate bucket" and "Show missing values" enabled. Is that working for you?

ch-bas commented 3 years ago

Sorry for this inconvenient behaviour of the plugin. I tested now the new changes with Kibana's version 7.10.2 and I couldn't reproduce it. Please feel free to report any new issue and I will try to take a look as soon as I find some free time, your contribution is much appreciated :) .

mmguero commented 3 years ago

I apparently hadn't grabbed the commits you made on the day you commented. My apologize. The missing/other buckets function is working now. Thanks so much!