search

observablehq / framework

A static site generator for data apps, dashboards, reports, and more. Observable Framework combines JavaScript on the front-end for interactive graphics with any language on the back-end for data analysis.
https://observablehq.com/framework/
ISC License
2.41k stars 110 forks source link

Data loader elasticsearch #1490

Closed walterra closed 3 months ago

walterra commented 3 months ago

As discussed on your Community Slack, I'd like to contribute a data loader example for Elasticsearch.

The original repo I created this in can be found here: https://github.com/walterra/observable-framework-data-loader-elasticsearch

A live view can be found here: https://walterra.observablehq.cloud/framework-example-loader-elasticsearch/

This Observable Framework example demonstrates how to write a TypeScript data loader that runs a query on Elasticsearch using the Elasticsearch Node.js client. The data loader lives in src/data/kibana_sample_data_logs.csv.ts and uses the helper src/data/es_client.ts.

Similar to #1477, I added the output of the data loader as a static file too.

To fully reproduce the example, you need to have a setup with both Elasticsearch and Kibana running to create the sample data. Here's how to set up both on macOS:

# Download and run Elasticsearch
curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.1-darwin-x86_64.tar.gz
gunzip -c elasticsearch-8.14.1-darwin-x86_64.tar.gz | tar xopf -
cd elasticsearch-8.14.1
./bin/elasticsearch

# Next, in another terminal tab, download and run Kibana
curl -O https://artifacts.elastic.co/downloads/kibana/kibana-8.14.1-darwin-x86_64.tar.gz
gunzip -c kibana-8.14.1-darwin-x86_64.tar.gz | tar xopf -
cd kibana-8.14.1
./bin/kibana

The commands for both will output instructions how to finish the setup with security enabled. Once you have both running, you can create the sample data in Kibana via this URL: http://localhost:5601/app/home#/tutorial_directory/sampleData

Finally, create the .env file with the credentials shared for the user elastic that were logged when starting Elasticsearch like this. To get the CA fingerprint for the config, run the following command from the directory you started installing Elasticsearch:

openssl x509 -fingerprint -sha256 -noout -in ./elasticsearch-8.14.1/config/certs/http_ca.crt
ES_NODE="https://elastic:<PASSWORD>@localhost:9200"
ES_CA_FINGERPRINT="<CA_FINGERPRINT>"
ES_UNSAFE_TLS_REJECT_UNAUTHORIZED="FALSE"
walterra commented 3 months ago

Thanks for the initial feedback, I updated index.md with the suggestions. Also added a link to examples/README.md.

Fil commented 3 months ago

Thanks for the very detailed procedure — I was able to test it locally.

I now have just a few more suggestions:

Another note in passing (but probably not actionable): as I was testing the setup, I was blocked as the enrollment string linked to IP address 10.100.0.2. But this address would not connect, and this prevented the installation from working at all. I realized I had to disable NordVPN, delete the whole es installation, and start again from scratch… and it worked. (I guess it's more an issue with NordVPN than with ES, but figured maybe you'd want to investigate this.)

walterra commented 3 months ago

Great you got it working with the instructions! Getting the SSL setup right can be a bit painful.

I updated the observable framework package to latest, added quotes to the .env part and made sure to mention the name of the dataset in the Kibana UI in b2b4d06.

About ES_CA_FINGERPRINT: Did the lowercase version without colons work for you? When I used it, I got this error message: ConnectionError: Server certificate CA fingerprint does not match the value configured in caFingerprint. That's why I added the openssl command then. I will check with our ES+clients team if we can improve that. I'll also forward your problem with NordVPN!

Update: We now have an issue to track the problem with the different fingerprint formats: https://github.com/elastic/elastic-transport-js/issues/107

Fil commented 3 months ago

Did the lowercase version without colons work for you?

Nope :). To clarify, it would have been great to have the correct format in that place and not have to do the call to openssl, as this is always stressful—but it's not a blocker for this PR!

walterra commented 3 months ago

I guess because the branch is in my fork you cannot push to it. I commited the quote updates!

Thanks for the deployment! Is there something I need to add so it gets the header with "Observable Framework" title and "View source" link? I can see it in the hello world example but not on the notebook you deployed for this data loader.

Fil commented 3 months ago

hmm no, that was my oversight; thanks for reminding me!

To deploy the examples we have to include the common configuration file which adds the specific headers:

yarn deploy -c ../observablehq.config.js