Ingest via file upload

[sergiimk]

Scope

• Limit file size to ~50MiB
• The ingestion flow will be done fully synchronously
  • Future phases will use the flow system
• Use cases to support:
  • Push ingest via file
  • Root dataset creation via file
  • Endpoint to perform schema inference on file (by URL)

Tasks

Notes:

• Pre-signed URLs are not fully good, there is no way to limit file size, but may work as a prototype
• Tagging to mark objects as temporary are not needed, as dedicated bucket would only contain temp files

Backend:

• File uploads with S3:

  • Provision separate S3 bucket with Terraform (kamu-data/kamu-infra#162, kamu-data/kamu-infra#165)
  • Kubernetes infrastructure should support binding the bucket and API server via configuration (helm-charts#27, kamu-deploy#95)
  • Backend should allow generating upload links for logged users only (kamu-data/kamu-cli#631)

• File uploads with local FS (kamu-data/kamu-cli#632)

• API server should support both cases of S3/localFS (kamu-data/kamu-node#82)

• Continuations after upload:

  • improve existing ingest API endpoint to use the uploaded file

Frontend:

• Dataset creation
  • Place for drag&drop / file upload initiation
  • on dataset overview page
  • during dataset creation flow
  • UI gets the upload URL and form fields for upload from the backend
  • UI uploads the file using given data, and obtains a file reference (URL+key)
  • UI calls schema inference endpoint and pre-populates push source creation page
  • Upon confirm it calls (synchronously) push ingest flow endpoint
• Push ingest
  • Place for drag&drop / file upload initiation
  • UI gets the upload URL and form fields for upload from the backend
  • UI uploads the file using given data, and obtains a file reference (URL+key)
  • UI calls schema inference + push schema validation endpoint(s) (and displays any errors)
  • Upon confirm it calls (synchronously) push ingest flow endpoint

Push ingest via file mechanics (OUTDATED - REVISIT)

We already have a REST endpoint that allows push small batches of data.

This ticket is to support file-based "push" for moderate (<200 MiB) volumes of data.

This includes:

• New endpoint to request URL for temporary file upload
  • Returns some assigned file ID and URL the client needs to use
  • In case of S3 repository:
  • Presigned upload URL
  • Sets temporary tags for the file
  • Set Up an S3 policy to clean such files after 24h
  • In case of local repository:
  • We need a local upload URL, unavailable in S3 mode (e.g. http://localhost:8080/upload/<file_id>)
  • File may be placed somewhere in the local cache
  • Cleanup will be performed by gc command or right after a successful push
• New endpoint to ingest data from a pre-cached file_id
• Store media type of the file somewhere along the way

[zaychenko-sergei]

@sergiimk Ready for acceptance

[sergiimk]

I suggest the following small improvements to be made before marking this epic completed:

• https://github.com/kamu-data/kamu-node/issues/98
• https://github.com/kamu-data/kamu-web-ui/issues/350
• https://github.com/kamu-data/kamu-node/issues/99