becomes very slow with many small files

[wzrdtales]

Is it normal that ltfs becomes incredibly slow when it stores many "smaller" files? The ltfs process when it is syncing the index (i guess) becomes busy on one CPU core (didn't disable multithreading but this seems to be single-threaded) for good 1 to 3 minutes (blocking reads and writes in that time) and also uses roughly 3GB RAM. Same when remounting a tape, it takes forever (good 5 to 10 minutes) for ltfs to load everything. Is this intended to be like that, is there a way to speed this up? Loading 3gb of data into ram shouldn't take that long

[piste-jp]

I'm not sure how many files are stored into on the LTFS tape. But it is expected and as designed.

LTFS stores every metadata on memory while it hosts the file system access. The entire metadata is stored on tape at periodical sync and unmount by XML. And the entire metadata is parsed at mount operation.

Creating and parsing XML data cannot be done by multi-thread.

All filesystem operation is blocked while sync operation is running to avoid metadata-change while sync operation.

I recommend to use the LTFS tape within 1 million files.

[wzrdtales]

thanks, that is helpful. I guess there are no plans to use a better format than XML in the future?

[piste-jp]

I guess there are no plans to use a better format than XML in the future?

We are using XML because it is defined by the LTFS format spec. And objective of this project is being the reference implementation of the LTFS format Specifications (See README).

Do you have any candidate about "a better format" ? We can propose it to SNIA TWG if you have candidates on your mind and it is reasonable.

In my sense, XML is the best choice because we can have a stable writer/parser library for XML (libxml2).

[piste-jp]

The ltfs process when it is syncing the index (i guess) becomes busy on one CPU core (didn't disable multithreading but this seems to be single-threaded) for good 1 to 3 minutes (blocking reads and writes in that time) and also uses roughly 3GB RAM.

You can use -o sync_type option to configure the sync period. See https://github.com/LinearTapeFileSystem/ltfs/wiki/ltfs.

[wzrdtales]

Do you have any candidate about "a better format" ?

Literally everything, XML is notorious for being slow, so no wonder this is a problem. I would need to dig into the requirements first to make a proper suggestion.

A few questions that come to my head: That XML contains basically only pointer information right?

Since the data is only synced, there are actually a lot of options beyond simple file formats. That leaves scope for simple databases like SQLite or also LevelDB, which would save the time implementing an sufficiently efficient lookup. If those are not an option, or not appropriate and it's a simple schema that needs flexible extension NDJSON (which has fast and multi threading capable parsers) might be target, but also just a straightforward dumped struct. But again, I would need to understand the requirements first so these are just loose thoughts.

[piste-jp]

A few questions that come to my head: That XML contains basically only pointer information right?

The index contains entire directory tree information and meta-data of all files and directories. And it also contains information of index itself.

Please refer the LTFS format spec in detail. The links are listed here.

[piste-jp]

Since the data is only synced, there are actually a lot of options beyond simple file formats. That leaves scope for simple databases like SQLite or also LevelDB, which would save the time implementing an sufficiently efficient lookup. If those are not an option, or not appropriate and it's a simple schema that needs flexible extension NDJSON (which has fast and multi threading capable parsers) might be target, but also just a straightforward dumped struct. But again, I would need to understand the requirements first so these are just loose thoughts.

The most important philosophy of LTFS format design is human readable. So binary format might not be acceptable in the TWG.

One reason of using XML is we can detect truncated index correctly. At that time, XML parser reports an error and LTFS can make a recovery operation against the tape.

For multi-thread processing, the tape drive is streaming device so we need serialize the result anyway. I'm not sure it make the process fast. LTFS needs to write down the data with 512KB block serially.

JSON (not NDJSON) might be better choice but we cannot find any good library (writer and parser) to use. Our first target is RHEL8, we need to pick the stable library only from RHEL8 official repository. We don't want to use EPEL or other 3rd party repositories.

[wzrdtales]

For multi-thread processing, the tape drive is streaming device so we need serialize the result anyway. I'm not sure it make the process fast. LTFS needs to write down the data with 512KB block serially.

The streaming doesn't really matter much in that regard, the operations usually do not really tend to do operations like read the end of the file and the beginning of the file, so that is probably not an issue. And for the dbs, they would be shut down, so wont be affecting this as well, but are ruled out from what you've written already.

One reason of using XML is we can detect truncated index correctly. At that time, XML parser reports an error and LTFS can make a recovery operation against the tape.

Ok, so this was made with human intervention in mind, that doesn't rule out all binary formats, per se, but definitely any of the database options.

JSON (not NDJSON) might be better choice but we cannot find any good library (writer and parser) to use.

I don't agree, if efficiency in parsing is a goal NDJSON is superior.

Our first target is RHEL8, we need to pick the stable library only from RHEL8 official repository.

Ok, sounds a little weird to me to develop only for RHEL8, but this is not my project o/c. What speak against compiling one of the libraries yourself and delivering it? I.e. https://github.com/json-parser/json-parser (normal json)

[piste-jp]

Our first target is RHEL8, we need to pick the stable library only from RHEL8 official repository.

Ok, sounds a little weird to me to develop only for RHEL8, but this is not my project o/c. What speak against compiling one of the libraries yourself and delivering it? I.e. https://github.com/json-parser/json-parser (normal json)

As you said, it is little little weird for "pure" open source project.

Actually this project is the subset of the IBM product (and I'm an IBMer). This code is almost as same as IBM Spectrum Archive SDE and a part of IBM Spectrum Archive LE. And we (IBM) takes care the customer who cannot access to internet easily. Most of tape customers have such kind of environment at this time. So we want to choose only libraries which is provided by RHEL8 DVD.

[piste-jp]

I don't agree, if efficiency in parsing is a goal NDJSON is superior.

In my understanding, it is little bit tricky to have a tree structure in NDJSON. It means if we need to have tree structure and truncated index detection feature, it is almost same as JSON.

Is it correct?

[wzrdtales]

So we want to choose only libraries which is provided by RHEL8 DVD.

But don't you need to deliver your own binaries to them? Still don't see the problem. But again I don't want to discuss this, this is your decision in the end.

What you've given me as information is already plenty enough to look into options ourselves. Should we fork and go into this on our own I will let you know of course.

[wzrdtales]

In my understanding, it is little bit tricky to have a tree structure in NDJSON.

That depends, if you represent the tree as an sorted array it is not an issue. If it is a gigantic object, then yes NDJSON wont help much since everything would be single record anyways.

[wzrdtales]

One more question:

I have seen that the HPE Version of this is actually outdated (2.4.0), but the latest 2.5.1 is already comparatively old. Is HPE still doing their own thing or has everything converged now to this repo?

[piste-jp]

I think first 2 digits of the version number shows the supported format version.

At this time, only 2.4 is supported on both IBM and HP, I believe. Version 2.4.5.0 is the latest in IBM version. I'm not aware which is the latest version in HP.

Now the master branch of this project is targeting the LTFS format spec 2.5. But we didn't implement all improvements yet. And we didn't release any version about 2.5. See here .

[wzrdtales]

Ic, thank you :)

[piste-jp]

I would like to close this once.

Please feel free to reopen if you have something.