[ML] Data Import Enhancement via Data Visualizer

[Abdhuraf01]

Overview

When it comes to importing data into Elastic Stack, we’re fortunate to have a variety of methods at our disposal, each suited for different scenarios. Whether leveraging Logstash for real-time data pipelines, Beats for lightweight data shippers, Elastic Agent for unified data collection, or even the Elasticsearch API for custom integrations, Elastic Stack is flexible enough to accommodate diverse data ingestion needs.

In one of my recent test cases scenario, I needed to import a structured dataset into the Elastic Stack to maintain historical records, and that too maintaining it manually. Since I wanted to keep it very simple, I used the Data Visualizer > Upload Data from a file option, which is incredibly efficient. This method automatically takes care of creating essential components like the index, ingest pipeline, and other settings while still allowing for customization as needed, as shown below.

Problem Statement

Here’s where things got interesting and a bit challenging. I realized that while the above approach works perfectly for an initial upload, it lacks flexibility for importing additional batches of data to the same destination. Let’s say I have another set of data (assuming I missed a portion or a similar set of data collected on a different day) that I want to add to the same existing dataset without creating a new index or data view since I wanted to look at all of them holistically; ideally, I’d like to simply append this data to my existing table. Right now, there’s no direct way to do this in Stack. Each time I import a new batch, it creates a fresh index and a new data view, which can quickly become unwieldy and cluttered.

This leads to my proposal: why not enhance the Data Visualizer with a straightforward "Append Data" feature? (as shown in the below image)

Feature Proposal

Introduce an "Append Data" option within the Data Visualizer. Imagine an option that lets users pick an existing index or data view and directly append their new data, provided it matches the schema. This would prevent redundant index creation, keep our data organized, and ultimately streamline the data ingestion process.

Such a feature would simplify data imports for cases where users are dealing with ongoing data additions, offering a cleaner, more manageable approach to growing datasets in Elastic Stack.

Hoping that this finds its way into the Product Enhancement roadmap soon!

---

cc. @stiltz

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[peteharverson]

Thanks for this enhancement request @Abdhuraf01 . We have actually started looking into this already.

See also https://github.com/elastic/kibana/issues/49159 and https://github.com/elastic/kibana/issues/31380.

[Abdhuraf01]

@peteharverson thanks for the update. The issues seems to be logged back in 2019 and it is not incorporated yet, so is this something in the nearer roadmap?

[peteharverson]

@Abdhuraf01 - yes this is on the roadmap and is currently being worked on.