[Question] Is there possible to construct Innodb file using Java in parallel ?

[pan3793]

I know this question may out of scope of this project. But I just wanna know is there possible to construct Innodb file using Java in parallel? If true, does that means it's possible to leverage some parallel engine like Spark to implement Innodb bulk insert rather than insert data using JDBC?

[neoremind]

It could be very difficult to build InnoDB data file directly, even we follow the format and build one, there are many internal values other than INDEX page to construct, like values inside FSPHDR, IBUF and INODE pages, without them, the file is corrupt and cannot be updated by mysql-server. I think your idea is very cool, but currently I cannot think of any possible solutions.

[pan3793]

Thanks for your explanation, now I know it's hard to build a full Innodb file.

there are many internal values other than INDEX page to construct, like values inside FSPHDR, IBUF and INODE pages, without them, the file is corrupt and cannot be updated by mysql-server.

Can I understand this sentence as it seems easier if we don't consider updating data, just build a read-only data file? Of course, it should be recognized by mysql-server.

[neoremind]

Hmm, good question. If you intend to make it a read-only file, it is possible to construct ibd file manually as long as you follow the row format. However, as the project naming specified, it is a "reader", in order to be a "writer", there are many works ahead, among the critical paths, for example, how to implement a B+ tree writer, how to handle page split and make it balanced, how to write TEXT blob page, etc. The devil is in the details.

[pan3793]

Sounds build a read-only file is simpler but still challenging. Really appreciate your guidance, it's really an amazing project and please don't concern, it's not a feature request, just a question about feasibility.

[neoremind]

Thanks for the good question as well! 😄