Integrating Data Lake Tables

This repository contains the supplementary materials and implementation codes for our paper Integrating Data Lake Tables (ALITE), accepted for VLDB 2023. You can find the technical report here.

Authors: Aamod Khatiwada, Roee Shraga, Wolfgang Gatterbauer, Renée J. Miller

Abstract

Over the last decade, we have made tremendous strides in providing tools for data scientists to discover new tables useful for their tasks. But despite these advances, the proper integration of discovered tables has been under-explored. An interesting semantics for integration, called Full Disjunction, was proposed in the 1980’s, but there has been little advancement in using Full Disjunction for data science to integrate tables culled from data lakes. We provide ALITE, the first proposal for scalable integration of tables that may have been discovered using join, union or related table search. We show that ALITE can outperform (both theoretically and empirically) previous algorithms for computing the Full Disjunction. ALITE relaxes previous assumptions that tables share a common attribute names (which completely determine the join columns), are complete (without null values), and have acyclic join patterns. To evaluate our work, we develop and share three new benchmarks for integration that use real data lake tables.

Repository Organization

codes folder contains ALITE and baseline source codes. It also contains the folders for each benchmark and for embedding given by each method.
statistics folder contains the statistics of benchmarks and the time taken to integrate tables on each benchmark using different techniques.
updated-alite-technical-report.pdf is the technical report for ALITE.
README.md file explains the repository.
requirements.txt file contains necessary packages to run the project.
synthesized_complex_schema.pdf file shows the synthesized complex schema used in the experiment.

Benchmark

Please visit this link to download Align Benchmark, Real Benchmark, Join Benchmark and the samples of IMDB Benchmark used in the experiments. The original IMDB benchmark is available at https://datasets.imdbws.com/.

Setup

Clone the repo

CD to the repo directory. Create and activate a virtual environment for this project

On macOS or Linux:

python3 -m venv env
source env/bin/activate
which python

On windows:

python -m venv env
.\env\Scripts\activate.bat
where.exe python

Install necessary packages. We recommend using python version 3.7 or higher.
```
pip install -r requirements.txt
```

Reproducibility

Download benchmarks and embeddings from this link and upload them to the codes folder. For convenience, you can run the following commands on your terminal which is based on gdown package. As the first command takes you to codes folder before downloading the files, make sure that you are in home of the repo.
```
cd codes && gdown --folder https://drive.google.com/drive/folders/1yUgL8TjQievzp8zvmHLpa_ClNzc5mTmD
```
```
cd Integrating\ Data\ Lake\ Tables\ / && unzip "*.zip" && rm *.zip && mv * ../ && cd .. && rm -r Integrating\ Data\ Lake\ Tables\ /
```
Run align_integration_ids.py to run the clustering algorithm that assigns the integration ids.
Run align_fd.py to compute full disjunction using ALITE.
Run pdelay_fd.py to compute full disjunction using BICOMNLOJ.
Run para_fd.py to compute full disjunction using ParaFD. Note that this algorithm can be used only for the tables having functional relationship.

Citation

@article{DBLP:journals/pvldb/KhatiwadaSGM22,
  author    = {Aamod Khatiwada and
               Roee Shraga and
               Wolfgang Gatterbauer and
               Ren{\'{e}}e J. Miller},
  title     = {Integrating Data Lake Tables},
  journal   = {Proc. {VLDB} Endow.},
  volume    = {16},
  number    = {4},
  pages     = {932--945},
  year      = {2022},
  doi       = {https://doi.org/10.14778/3574245.3574274},
}

northeastern-datalab / alite

readme