JackKelly light-speed-io issues

JackKelly / light-speed-io

Read & decompress many chunks of files at high speed

MIT License

42 stars 0 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Multi-dataset abstraction layer

#142 JackKelly opened 1 week ago
2
Utility to extract, reshape, and store data for a subset of the data. e.g. for extracting timeseries for single PV sites from gridded NWPs

#141 JackKelly opened 1 week ago
0
Generic `user_data` type for `Chunk`

#140 JackKelly opened 1 month ago
0
Remove `crates /lsio_uring/benches`

#139 JackKelly opened 1 month ago
0
Allow users to provide `filesize` etc. using suffixes like `m` and `k`

#138 JackKelly opened 1 month ago
0
Why does `lsio_bench` give insanely high bandwidth numbers when reading a single chunk per file?

#137 JackKelly opened 1 month ago
0
MVP `lsio_benchmark`

#136 JackKelly closed 1 month ago
1
Benchmark `lsio_uring`

#135 JackKelly closed 1 month ago
1
Maybe the `HIGH_WATER_MARK` idea doesn't make sense?!

#134 JackKelly opened 1 month ago
0
Stop using iterations (in GitHub). Instead use milestones.

#133 JackKelly closed 1 month ago
1
Maybe use `Bend` to parallelise compute

#132 JackKelly opened 1 month ago
0
Keep track of SQ entries, in a 1:1 relationship

#131 JackKelly opened 1 month ago
0
Build a command-line benchmarking app

#130 JackKelly opened 1 month ago
3
Calling `uring.submitter().submit()` on every loop is probably expensive

#129 JackKelly opened 1 month ago
0
Test new io_uring code

#128 JackKelly closed 1 month ago
0
Benchmark for new io_uring code

#127 JackKelly opened 1 month ago
0
Implement optimisation of byte ranges

#126 JackKelly opened 2 months ago
0
Implement the simplest io_uring "worker" using threads & `crossbeam::deque`

#125 JackKelly closed 1 month ago
0
Stop/Start: Enable LSIO to be gracefully stopped & started (e.g. in response to carbon intensity)

#124 JackKelly opened 2 months ago
0
Benchmark `lsio_uring` vs `object_store_bridge`

#123 JackKelly opened 2 months ago
0
Implement the simplest io_uring "worker" using threads & `crossbeam::deque`

#122 JackKelly closed 1 month ago
10
Plan to re-write LSIO

#121 JackKelly closed 2 months ago
1
Document decisions taken

#120 JackKelly opened 3 months ago
0
Rigorous benchmarking

#119 JackKelly opened 3 months ago
0
Rigorous automated testing

#118 JackKelly opened 3 months ago
0
`Typestate`: Use a custom type for each state (for each `IoOperation`) & use Rust's type system to make sure we only progress forwards

#117 JackKelly closed 3 months ago
4
Create a `struct UserData` to manage how we encode uring `user_data`

#116 JackKelly closed 2 months ago
1
`AlignedBuffer` should use `valid_slice` to define the `start` and `len` that the user is interested in

#115 JackKelly closed 2 months ago
0
Allow `AlignedBuffer` to return "owned slices" (like `Bytes`), which refer to the same buffer in RAM

#114 JackKelly closed 2 months ago
6
Check if `ndarray`, `numpy`, & compression crates can consume `&[u8]` (and hence consume from `AlignedBuffer`)

#113 JackKelly opened 3 months ago
1
Maybe use `Bytes` within `AlignedBuffer`

#112 JackKelly closed 3 months ago
2
New design doc

#111 JackKelly closed 3 months ago
0
Maybe change name (away from LSIO)

#110 JackKelly opened 3 months ago
15
New design sketch for March 2024

#109 JackKelly closed 2 months ago
1
Implement `AlignedBuffer::into_box`

#108 JackKelly closed 3 months ago
2
New crate: `lsio_object_store_bridge` to perform IO using `object_store`

#107 JackKelly opened 3 months ago
0
New crate: Provide iterator-like methods on a crossbeam channel

#106 JackKelly closed 3 months ago
2
How to identify each returned `Item`?

#105 JackKelly closed 2 months ago
1
What would user's code would look like the for simple use-cases? (After dropping `async`, and instead only using `channels` and `Rayon`)?

#104 JackKelly opened 3 months ago
9
look into using Rayon for the IO thread pool. Can we associate an `IoUring` instance with each thread? Can we publish tasks to other threads?

#103 JackKelly closed 3 months ago
3
New crate: `lsio_optimise_io_plan` (e.g. merge nearby operations; split large ops)

#102 JackKelly closed 3 months ago
1
Check the number of bytes read

#101 JackKelly closed 3 months ago
0
Handle case when `read` actually reads less data than the user requested

#100 JackKelly opened 3 months ago
0
Splits reads of more than 2 GiBytes into multiple reads

#99 JackKelly opened 3 months ago
0
Write unit-tests for `AlignedBuffer` to check behaviour when using `start_offset`.

#98 JackKelly opened 3 months ago
0
Ensure that the `read` operation reads the correct number of bytes

#97 JackKelly closed 3 months ago
0
Remove get

#96 JackKelly closed 3 months ago
0
Why's the io_uring code so slow?! :slightly_smiling_face:

#95 JackKelly closed 3 months ago
5
Use a flat crate structure so this git repo can store multiple (interconnected) crates in a single workspace

#94 JackKelly closed 2 months ago
0
Drop `ObjectStore` & `async`/`await`. Use Channels instead. Focus entirely (for now) on io_uring for local file storage.

#93 JackKelly closed 2 months ago
1