lundmark
commented
5 years ago Tons of good feedback in here! I really think we should start fixing / cleaning up the readme and these are all really good suggestions in order to improve that.
Here's a bit of a walkthrough that probably explains some of the concepts used here.
Terms: Type library: a dataset that describes the types, unions and enums and information regarding those so that the data library understands the data that it serializes. The type library can be in text format (which is usually how you write them) and binary format. The "tlc" (type library compiler) converts the text format into a binary format, and also has the option to generate a c/cpp-header that your application can include in order to get the data structs that DL operates on.
Packed instance: An instance of data, packed into a chunk that is prepared for the target platform. This means that endian swaps, pointer size etc is taken into account and is prepared for the platform. When you read in a packed instance on your target platform, the only thing DL needs to do is clean up pointers (which are stored as offsets when packed).
So DL supports the following example pipeline: typelibrary json -> tlc.exe -> typelibrary binary file + c/cpp header file data instance json -> dl_pack -> packed instance, binary format file
You can then use dl to: include the generated c/cpp typelibrary file typelibrary json/binary -> dl_context_load_txt_type_library() / dl_context_load_type_library() packed instance -> dl_instance_load() / dl_instance_load_inplace() -> useable memory format for your platform.
Now, hypothetically there are a lot of other things you can do / use dl to. You don't have to generate the c/cpp header, and instead you can use dl_reflect to walk over the data (not really fast or optimal tho), and a bunch of other things. For example we use the type library so that our editor understands how to represent the data in property editors.
To answer some of your specific questions: Yes DL seems to null-terminate all strings (fredrik will have to correct me here if I'm wrong). load vs store: load in-memory-instance from packed format / store in-memory-instance to packed format pack/unpack: pack txt/json instance to packed format / unpack from packed format to txt/json. Difference vs variable / static sized:
int my_arr[16]; // static sized array
struct {
int *data;
int count;
}; // variable sized arrayI hope this straightens out some of the questions and what the library does and how it works! Now, let's start looking at the readme and make sure that people more easily can understand and adapt the library!
wc-duck
RandyGaul
Data Library looks like a nice library! But there are some severe problems with the build system and docs which have prevented me from trying the library out after dedicating multiple hours of time. I'm just writing down all my thoughts as feedback. I am quite interested in trying out Data Library, but these issues are blockers for me personally.
I've never seen bam before and have no idea how to build Data Library.
I tried:
appveyor.ymlAfter this point I spent a couple hours poking around and am giving up. Why not provide a cmake script, or pre-built binaries? I don't know anyone that has heard of bam before, and using it as the only way to build the code locks out many potential users (like myself).
I also noticed there is no full example demonstrating the entire workflow of Data Library. When a new user comes into the repository here are the steps they will look for the in the documentation:
example.binin the current docs? That's confusing. I would expect something more likeexample_typelibrary.binThe docs are entirely missing step 1. Step 2 is already a good example, but it does not match the rest of the steps, which is confusing. Step 4 is important for people to get a feel of the workflow. Some notes one what is expected in the output files is really important. At this point I've already invested a couple hours into researching Data Library and still have no idea what the output of dltlc is going to look like.
load = memcpy + with plus-menu guaratee complete data when built error-checks platform specific data ( convertion between platforms avaliable on all platforms )This phrase really confused me. It took at least five minutes to read (over and over again), and I still don't quite understand what it is saying. Here is my best attempt at a revision, but I still don't know what
plus-menu guarateemeans.Here's my edit:
The parts section of the readme can also use some tuning. The information needs to be sorted by the user workflow, and listed out in-order of the workflow from a user's point of view. Here's my attempt at an edit:
What is dl_pack? There is no documentation for this.
The entire readme should be sorted by priority. The first thing users will want to see is how to build Data Library. Ideally there should be prebuilt binaries for common platforms available as release packages through github releases.
The next thing would be the workflow.
After this the list of supported types can be shown. This should actually be a list instead of a big paragraph. Here's a less confusing list:
Supported types.
Unsupported types considered for future releases.
The section called
TLD ( Type Library Definition ) format:is not very helpful. I have almost no idea what they are trying to say. For example"module"-section-- okay now I know that "module" is asection, but I don't know what a section is at this point. The docs have not explainedsection. The same goes for"members"-array, and why is"type"all alone without a dash like the other ones?I have the same feedback for the section
Text data format:.For more in deapth examples, see EXAMPLE-file- there is no example file. Where did it go?