search

BlazingDB / blazingsql

BlazingSQL is a lightweight, GPU accelerated, SQL engine for Python. Built on RAPIDS cuDF.
https://blazingsql.com
Apache License 2.0
1.93k stars 183 forks source link

Support for path-style access to s3 buckets #1558

Open marcnc27 opened 3 years ago

marcnc27 commented 3 years ago

Blazing supports AWS s3 and MinIO by overriding the endpoint. Thus, with non virtual host style bucket access it seems blazing context can not load a table from s3.

Possible solutions

  1. Support s3 client overriding (e.g. using s3fs as a filesystem params like in pandas)
  2. Support extra parameters to specify path-style access or virtual host style access
  3. Allow building blazing tables using a file descriptor's like object (emancipating from s3 adherence)

----For BlazingSQL Developers---- The feature could be implemented directly in table creation methods as a parameter or set of parameters

wmalpica commented 3 years ago

We will be embarking on a filesystem plugin system refactoring project in the near future. We will keep in mind this issue when we do so. Thank you for bringing this issue to our attention