Update S3 data architecture and integration - Githubissues

FormalLanguageConstrainedPathQuerying / CFPQ_Data

Graphs and grammars for Context-Free Path Querying algorithms evaluation.

https://formallanguageconstrainedpathquerying.github.io/CFPQ_Data/

Other

10 stars 15 forks source link

Update S3 data architecture and integration #63

Closed vadyushkins closed 2 years ago

vadyushkins commented 3 years ago

Proposal

The data architecture at the moment looks like this:

1.0.0 -- Release version of cfpq_data
- MemoryAliases
- graphs -- Separate rdf files, each packed in its own .tar.gz archive
- RDF
- graphs -- Separate rdf files, each packed in its own .tar.gz archive

I suggest to do the following data architecture:

Release version of cfpq_data
- csv file for each graph

With CSV headers [from,label,to]

Pros

The proposed architecture simplifies the manipulation of graphs in S3
Loading will be faster
CSV is simpler and more convenient to represent labeled graphs than RDF

Cons

It may take more disk space than .tar.gz archives
CSV format is not as rich as RDF
No other data format provided

vadyushkins commented 2 years ago

Closed by #66