search

vaexio / vaex

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
https://vaex.io
MIT License
8.31k stars 591 forks source link

[Locking] When converting tsv files, it would be nice to specify the directory where the vaex lock files go #1689

Closed mwang87 closed 2 years ago

mwang87 commented 3 years ago

Description When converting tsv files, it would be nice to specify the directory where the vaex lock files go. If we could create an option for example: 

vx.from_csv(path_to_file, lock_folder="/tmp")

that could make this clutter a bit better also, sometimes the current working directory is read only even though the location for the HDF5 and tsv files are writeable.

Is your feature request related to a problem? Please describe. Currently its in the active working directory which might clutter things and is not actually cleaned up.

maartenbreddels commented 3 years ago

I agree, I initially though that lock files would be removed, but due to possible race conditions on Unix systems they cannot. I think /tmp/vaex is a good default for that

maartenbreddels commented 2 years ago

See https://github.com/vaexio/vaex/pull/1797

mwang87 commented 2 years ago

Thanks @maartenbreddels this is great, can't wait til next release!