search

quantcast / qfs

Quantcast File System
https://quantcast.atlassian.net
Apache License 2.0
643 stars 171 forks source link

reader can not read any data until the writer close the file #215

Open quantsword opened 7 years ago

quantsword commented 7 years ago

I run 2 programs in separate process. On program created a file and write to the end 64K a time till the file size reaches 1G. The other program was launched after writer started (file created) to read the same time. In my first iteration, reader program could not read anything even after write finishes and exit. I added "UpdateFilesize" to update filesize when read returns 0. Reader can read data after writer finishes.

I believe reader should be allowed to read even if write does not close the file yet. It is ok for writer to keep the lease on last chunk. But for completed chunk, It should allow everybody to read.

mckurt commented 7 years ago

Hi,

I replicated this setup: one writer and one reader running concurrently. As soon as a chunk got fully written, e.g. gets stable, reader started to read the chunk. However, my setup was using 1x replication without striping.

Mehmet

mikeov commented 7 years ago

Striped files, including RS, cannot be read until closed by design: close call sets the logical file size (EOF). Until striped file is closed its logical file size remains 0.

quantsword commented 7 years ago

Thanks for your response.

Rationale why this is needed. Scenario #1, It is typical for data processing pipeline to use files to connect the input and output of 2 process steps. Step 1 generate items and write to file, step 2 read this file and do further processing. If process2 is a map process (no need for sorting, shuffling, etc), it can start the process as soon as data is readable from the file. After it consume all the data items, it can wait till more data become ready.

Scenario #2, data is appended to a qfs file and index is created at the same time. Data request can come any time for any record. If an record is available as indicated in index (index can be local file instead of QFS), it will try to read from QFS file with the specified location. It can wait a little bit but not till QFS file close, as append will always happen.

My Question, after QFS collected 6 strips and generated 3 recovery strips, it will push these 9 strips into 9 chunk server. The write client can notify meta server with a updated file size. Is there any design concerns in this approach?

thanks