Open BenWoodworth opened 1 year ago
Another issue with the way things are done at the moment is related to change I mentioned on https://github.com/BenWoodworth/knbt/issues/13#issuecomment-1747769762.
This recent change in 23w40a will allow not only Compound Tags, but also String Tags to be encoded as root tags. If we were to generalize this, any type of tag could potentially be encoded as a root tag.
I don't think that should actually be an issue with the way I have it implemented now for this issue. You'd need a custom serializer since you can't e.g. annotate a string with @NbtNamed, but adding the annotation to the serializer descriptor like this should work:
object RootStringSerializer : KSerializer<String> {
override val descriptor: SerialDescriptor =
object : SerialDescriptor by PrimitiveSerialDescriptor("RootStringSerializer", PrimitiveKind.STRING) {
@ExperimentalSerializationApi
override val annotations: List<Annotation> = listOf(NbtNamed("root string name"))
}
override fun serialize(encoder: Encoder, value: String): Unit =
encoder.encodeString(value)
override fun deserialize(decoder: Decoder): String =
decoder.decodeString()
}
What is interesting now though. I was debating between @NbtNamed
and @NbtRoot
and landed on @NbtNamed
. But with #36 being a thing now, I'm probably going to go with @NbtRoot
instead since that breaks my assumption about a named tag (which is a compound with one named entry according to the spec) and the root being the same thing.
Yes, the issue I'm trying to point out is that very assumption (named tags are compounds with one entry), which in my view at least has always been more of a convention, and not a hard limitation of the NBT format itself, even before the change in 23w40a. It's making very cumbersome to perform some operations that should've been trivial (writing a simple Stringtag should not require all that work mentioned on https://github.com/BenWoodworth/knbt/issues/29#issuecomment-1751852009).
I will open a new issue detailing everything (or possibly edit #36) as to not get too off-topic. The point I'm trying to make isn't really related to root class names, but as to how the root tag itself is structured/treated during serialization/deserialization.
As far as discussing root tags and how they're structured/treated (and the mental model for it in general), this issue's probably the best place to discuss that. (Instead of a new issue, since the structure/treatment is exactly what this issue is about).
And your issue for nameless root tags could be a discussion of how the new variant fits into that mental model. I have a few ideas after sitting on it yesterday, so I'll drop a comment over there too.
And I also 100% agree that it's cumbersome. The RootStringSerializer
example just serves to demonstrate it's possible with the one NbtNamed
line in the current encoding. And if needed, it can all be neatly wrapped into a general class from knbt, e.g. NbtNamedValue<String>("name", "value")
Awesome, I will gather my ideas and make a post here when I have some time then. Some of the points are still more aligned with #36, so I will post those there.
Partially in reply to your comment on #36:
As for the named tag, it was more of a suggestion. If you already considered it at some point, and still decided doing it another way, I don't see any point on insisting on it.
I've thought about this more than I'd like to admit (😅), wanting to find a solution for:
NbtTag
representation fully capture/mirror the serialized NBT with named root tagsAnd I had a couple misconceptions when I originally considered how names should be modeled (and in all my messages before this):
• Every NBT file will always implicitly be inside a tag compound, [...]
An NBT file consists of a single GZIPped Named Tag of type TAG_Compound
{"": {Data: {...}}}
(Which replaces the approach outlined in this issue's description)
@NbtName
annotation in serial descriptors
NbtName
will have an empty name by default@SerialName
, both with the annotation's naming, and with how a class's @NbtName
will be ignored when the class is serialized as a part of some other data@NbtNamed
, since everything will already be treated as named.NbtNamed
class below, since the annotation/class can't be named the sameNbtNamed<T>(name, data)
class to override the data
's name
data
's serializer, just adding/replacing its NbtName
annotation and changing its serial name, and delegating everything elseNbtTag
hierarchy to be root-name-aware for those who want it, potentially with encodeTo
/decodeFromNamedNbtTag()
functions(moved to the issue description)
A way to serialize a class without a root tag and just have everything be a flat compound tag would be nice. Using this in actual Minecraft modding I have run into many scenarios where I don't want a root tag because it gets embedded in another bit of nbt.
A way to serialize a class without a root tag and just have everything be a flat compound tag would be nice. Using this in actual Minecraft modding I have run into many scenarios where I don't want a root tag because it gets embedded in another bit of nbt.
I agree. Having like a encodeToByteArrayFlat
or encodeToByteArrayRaw
that would just encode the payload without the root tag would be useful.
Right now I do this
NBT.encodeToNbtTag(serializer, input).nbtCompound[serializer.descriptor.serialName]!!
It's kotlin, because I write my Minecraft mods in Kotlin, but having an encodeToNbtTagFlat
would be awesome
I also have some stuff for converting between the knbt representation and the Mojmap representation if anyone needs that:
This is also when I realized that knbt doesn't have an equivalent of Minecraft's EndTag
, so I had to use null. Unfortunately you can't have a null value in list and compound tags in knbt, so some data loss occurs
I also have some wrappers for using kotlinx serializers for Minecraft Codecs and Codecs as serializers
Probably not the best place to put this stuff, you should create a gist :)
If I understand it correctly, the root tag is a feature of the binary format for type checking (you can't decode without it)? If so, then only the binary format encoders/decoders (e.g. encodeToStream
) need to take the root tag into account (wrapping everything including primitives and NbtTag
in a root tag). Methods like encodeToNbtTag
or decodeFromString
that don't use the binary format could just do without the root tag. You could also have the flat
methods for binary format as mentioned earlier for encoding just the payload without the root tag. Decoding flat binary data would be unsupported.
Also are there any updates on the work? I'm open to doing some contributions.
Probably not the best place to put this stuff, you should create a gist :)
If I understand it correctly, the root tag is a feature of the binary format for type checking (you can't decode without it)? If so, then only the binary format encoders/decoders (e.g.
encodeToStream
) need to take the root tag into account (wrapping everything including primitives andNbtTag
in a root tag). Methods likeencodeToNbtTag
ordecodeFromString
that don't use the binary format could just do without the root tag. You could also have theflat
methods for binary format as mentioned earlier for encoding just the payload without the root tag. Decoding flat binary data would be unsupported.Also are there any updates on the work? I'm open to doing some contributions.
So, here's some background of the entire mess:
The root tag is just a name for the outer-most NBT tag. Although the spec lightly implies that it should be a Compound, the way it is written allows it to be any type in essence. The spec itself is a mess, and was probably not thought of thoroughly back when it was defined, considering the recent changes.
The spec defines NBT tags consisting of a name and its payload, with nameless tags being considered the edge-cases (such as the values in a nbt lists). However, it's easier if you think of things in reverse, that is, as named tags being the edge-case. That way, you leave names as being specific data of Compounds, pretty much just like how Json is structured (json objects contains key value pairs). Of course, the root tag becomes the outlier in this case, since it is also still named.
Up until recently, that is how things worked, but on 23w31a the root tag name was completely dropped when encoding data on the network side, since it was completely pointless (the root tag name was always empty in every single case on the Vanilla implementations). Not only that, changes were made so any type of NBT tag can be serialized as-is. These recent changes made it so that the NBT format is structurally identical to the Json format (compounds are objects, strings are text, lists are arrays, and everything can be serialized as-is without any wrapping).
Thanks for the input and use cases! I haven't read through all the new replies just yet, but I've got a new design mostly written up, and I'm going try to finish fleshing that out and update my last comment with it later today once I'm done.
As far as what needs to be done, this issue as originally stated has already been implemented, but it still has some problems and pain points that should be addressed in the new design I'll post. After that, it shouldn't be too much to implement, since I'm expecting it to be more of a refactor effort with some small API changes/additions afterwards, being that it's a change to how NBT's modeled. So I'm not sure there's a good way to contribute, but I've got more free time now after just finishing some at-home construction the past few months, and I'll be posting updates here!
v0.12 will have a good number of API changes addressing pain points of the library, and this issue is the last big thing in the way before I get to pushing it out the release. Possibly finishing up this issue within a month if all goes smoothly :)
Updated! I'll read through the comments tomorrow when I get a chance :)
@KP2048 (comment):
A way to serialize a class without a root tag and just have everything be a flat compound tag would be nice. [...]
@pandier (comment):
I agree. Having like a
encodeToByteArrayFlat
orencodeToByteArrayRaw
that would just encode the payload without the root tag would be useful.@KP2048 (comment):
Right now I do this
NBT.encodeToNbtTag(serializer, input).nbtCompound[serializer.descriptor.serialName]!!
[...], but having an
encodeToNbtTagFlat
would be awesome
Sorry for the late reply! The new design (in the issue description) gets rid of the root compound entirely, so encodeToNbtTag
/encodeToByteArray
/etc. work exactly the way you want. Nameless, "raw", and no root compound to un-nest.
Then because there's no root compound naming at all anymore, there's a new (and optional) NbtNamed("name", value)
for people who do still need the root name. For that, there will be encodeToNamedNbtTag
, and also using NbtNamed
with the existing encodeToByteArray
/etc. functions.
Basically all the functions you're using now will just work, without needing to add any additional logic to deal with names. I'm going to start implementing this soon, so let me know how it sounds or if you have any questions/concerns about the new way serializing will work :)
Sorry for the late reply! The new design (in the issue description) gets rid of the root compound entirely, so
encodeToNbtTag
/encodeToByteArray
/etc. work exactly the way you want. Nameless, "raw", and no root compound to un-nest.Then because there's no root compound naming at all anymore, there's a new (and optional)
NbtNamed("name", value)
for people who do still need the root name. For that, there will beencodeToNamedNbtTag
, and also usingNbtNamed
with the existingencodeToByteArray
/etc. functions.Basically all the functions you're using now will just work, without needing to add any additional logic to deal with names. I'm going to start implementing this soon, so let me know how it sounds or if you have any questions/concerns about the new way serializing will work :)
Awesome to hear that!
I do have a question regarding the new approach though. Maybe we can even use this discussion to iron things out.
Is your current idea:
NbtNamed
as a wrapper of sorts, with custom serialization logic to be handled internally by the encoder/decoder? In this case, the object could be passed directly to encodeToByteArray
/etc, without the need of a specific encodeToNamedNbtTag
method.encodeToNamedNbtTag
? In this case, the input of the method could be (name: String, payload: T)
, without the need for the NbtNamed
wrapper.It sounds kind of ambiguous to me at the moment.
Nice! I look forward to bumping my Gradle dependency
Nice job! Exactly what I wanted and need
@WinX64 (comment):
I do have a question regarding the new approach though. Maybe we can even use this discussion to iron things out. [...]
Good question! My current idea is mostly #1, with NbtNamed
being a way to change the name of the payload when encoding.
Here's what I have in mind for encodeToNamedNbtTag
:
fun <T> encodeToNamedNbtTag(serializer: KSerializer<T>, value: T): NbtNamed<NbtTag>
In this design, every serializable type is treated as having a NBT name (which root-named NBT variants use for the root value). This function is exactly the same as encodeToNbtTag
, except it also captures value
's NBT name. So this function gives a more complete in-memory representation of how value
is serialized, instead of missing the name.
And in general, I think of NbtNamed(name, value)
as effectively replacing its value
's NBT name. Unless a serializable type is (statically) annotated with @NbtName
, its name will be an empty string by default. And NbtNamed
offers a way to (dynamically) change a value's NBT name. And with the NbtNamed<NbtTag>
return type, it replaces NbtTag
's empty name with value
's.
Let me know if that clears things up a little!
Problem
Currently (v0.11) classes are serialized within a single-entry NbtCompound, with the serial name as the key. But, only if serialized at the root.
This behavior is convenient, but has a couple issues:
@SerialName
doesn't support empty strings, which is common in NBT files.NbtContentPolymorphicSerializer
, because extra nesting is added for a class depending on if it's at the root, looking at the shape of the tag alone isn't enough, and properly accounting for it is undesirable complexity.Nbt.encodeToNbtTag(someClass)
andencoder.encodeSerializableElement(serializer, someClass)
don't produce the same result.Old redesign
This issue's initial redesign has been implemented, but is being replaced by the new design proposed in this comment
Original solution
- Remove the special conditional logic for adding serial names at the root, so data serializes the same regardless of where in the serialization process/data it is - This makes deserializing based on the structure more straightforward - Add an annotation that, when applied to a serial descriptor, instructs the serializer unconditionally to nest the data in a single-entry `NbtCompound` - This covers the use case of top-level NBT binary/files needing to be a named tag (single-element NbtCompound) - From the docs: > An NBT file consists of a single GZIPped Named Tag of type TAG_Compound. - So `@NbtNamed` might be a good annotation name. `@NbtRoot` and `@NbtFile` have been used in earlier versions of knbt, but they seem too specific ### Tasks - ☑ `StructureKind.CLASS`/`OBJECT` - Covers the previous default `class` nesting using the serial name - ☐ `StructureKind.MAP` - ☐ `StructureKind.LIST`New Design
This design aims to add a concept of NBT names that all serializable types have, with every type either:
The initial implementation will mainly focus on implementing static naming, since most use cases don't need logic around the serialized NBT names, and it's also not clear yet how an API for dynamic names should behave. See the dynamic names section below for details.
For the scope of this initial design, the NBT name only applies to the root tag name. Though in the future, especially with dynamic names, it's possible the NBT name could apply more broadly. (e.g. a value within an NBT compound knowing its own element name while deserializing)
Use Cases
These are use cases that are being designed for in this new approach
Default use, without using any named NBT API
@NbtName("")
Json
is strict by default, e.g. withisLenient
andignoreUnknownKeys
ignoreNbtName
configuration option or similarStatically setting the NBT root name for a type
@NbtName
(or including it in the serial descriptor annotations)Inspecting NBT (including root name) by decoding to an in-memory representation
Inspecting how data is encoded to NBT (including root name) through an in-memory representation
Named-root NBT variants
Only some variants of NBT have root names encoded. With this new design, serializing values should be the same between named and unnamed root variants (instead of named variants being modeled as nested in an NBT compound).
Unnamed root variants:
net.minecraft.data.Main
NBT -> SNBT conversion tool, since its output SNBT doesn't include the root name anywhere)NbtTag
sStatic NBT names for all serializable types
@NbtName
annotation, on classes or in serial descriptorsDynamically serializing NBT names, and future encoding/decoding API
Add a basic
NbtNamed
class for an in-memory representation of the (root) NBT value and its name.T
, since the newer Java Network NBT supportsnull
(TAG_End
) as a root valueNbtNamed
's serializer will be specially handled by the NBT encoder/decoder for nowNbtNamed
's serializerNbtNamed
should override that value's NBT nameNbtNamed
(directly, or indirectly when delegating serializers) should give precedence to the outermost dynamic nameCare was taken when deciding how this would work, making sure there's room for full-blown dynamically serialized names to be added later in a forward-compatible way.
NbtNamed
at the root for nowTAG_Compound
entries as being a list of named tags, so there is an interpretation where it makes sense for NBT names` to be used with nested tag entries