search

zazuko / cube-creator

A tool to create RDF cubes from CSV files
GNU Affero General Public License v3.0
11 stars 2 forks source link

refactor: graph store post new dimension mappings #1407

Closed tpluscode closed 1 year ago

tpluscode commented 1 year ago

This changes how new dimension mappings are saved to the database when "auto-mapping"

The current implementation performed an INSERT with a complex pattern and a long list VALUES clause. Potentially thousands of items long.

Since the full set of these VALUES is loaded to memory anyway, they are actually cross-referenced in memory and new items are streamed directly as quads using the graph protocol. This appears much more database-friendly. While Stardog does cope, local fuseki actually failed to insert when mapping a large dimension of municipalities (3k terms)

changeset-bot[bot] commented 1 year ago

🦋 Changeset detected

Latest commit: 7350f75e9267cb07d80d829fa8050c60bcc589cb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package | Name | Type | | ---------------------- | ----- | | @cube-creator/core-api | Patch |

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

codecov-commenter commented 1 year ago

Codecov Report

Merging #1407 (b5d0984) into master (7884e0e) will decrease coverage by 3.26%. The diff coverage is 100.00%.

:exclamation: Current head b5d0984 differs from pull request most recent head 7350f75. Consider uploading reports for the commit 7350f75 to get more accurate results

@@            Coverage Diff             @@
##           master    #1407      +/-   ##
==========================================
- Coverage   80.77%   77.51%   -3.26%     
==========================================
  Files         196      196              
  Lines       13448    13466      +18     
  Branches      741      721      -20     
==========================================
- Hits        10862    10438     -424     
- Misses       2578     3020     +442     
  Partials        8        8
Impacted Files Coverage Δ
apis/core/lib/domain/queries/dimension-mappings.ts 100.00% <100.00%> (ø)

... and 13 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.