search

otland / forgottenserver

A free and open-source MMORPG server emulator written in C++
https://otland.net
GNU General Public License v2.0
1.56k stars 1.04k forks source link

Design a flatbuffers based map format #2206

Open djarek opened 7 years ago

djarek commented 7 years ago

Design goals

Other requirements

ranisalt commented 7 years ago

Can you provide an OTBM format specification for reference? I think what is currently specified in the source code is very lacking and too superficial about what can be contained in an OTBM file.

djarek commented 7 years ago

This is a very generic spec. Too bad the original author didn't write an RFC :D.

Escape = 0xFD
Start = 0xFE
End = 0xFF
NodeTag = Escape | Start | End
OTBM_ROOTV1 = 0x01
NodeType = OTBM_ROOTV1 | 0x02 | ... | 0x10
EscapedByte = (0xFD 0xFD) | (0xFD 0xFE) | (0xFD 0xFF)
UnescapedByte = 0x00 | 0x01 | ... | 0xFC
PropertyStream = {EscapedByte | UnescapedByte}
Node = Start NodeType [PropertyStream] {Node} End
Identifier = "OTBM" | "OTBI"
RootHeaderVersion = u32le
RootHeaderWidth = u16le
RootHeaderHeight = u16le
RootHeaderMajorItemsVersion = u32le
RootHeaderMinorItemsVersion = u32le
RootHeader = RootHeaderVersion RootHeaderWidth RootHeaderHeight RootHeaderMajorItemsVersion RootHeaderMinorItemsVersion
RootNode = Start OTBM_ROOTV1 {Node} End
OTBFile = Identifier RootNode
conde2 commented 7 years ago

Wouldn't worth rework OTB aswell when doing this changes ? It could totally be done in XML/LUA or anything more friendly than what we have now.

djarek commented 7 years ago

@conde2 If you mean the items file, I'm not sure why it's in binary format now, if there are no size issue, then it could be merged with items.xml. As for xml maps? Nope, not going to happen.

conde2 commented 7 years ago

Yes, I was talking about the items.otb, it would be awesome have it merged with items.XML. About the map I know it is not viable doing it in XML, i was strictly speaking about items.

ranisalt commented 7 years ago

Expect items.xml to grow orders of magnitude in size since binary formats can store much more info on a single byte compared to text formats.

Kamenuvol commented 7 years ago

I do not like items.xml format either. It's so repetitive. Why not use custom tags instead of attributes? An items.lua would be smaller size including everything otb and xml have, and it's easier to create custom attributes, except the ones we use binary shifting to join some of them.

djarek commented 7 years ago

TBH I don't consider a size increase of 3x or more to be acceptable. If migrating item information fully to XML is not feasible then we should consider using a simillar protobuf format for all item information as well.

Nu77 commented 7 years ago

OTB is a pain in the ass. Theres no cross-platform tools to handle it properly. About size maybe a module file like this could be useful:

<file name="781" path="items781.xml" />
<file name="854" path="items854.xml" />
<file name="960" path="items960.xml" />

I've used tibia versions as example since we use it as standard.

djarek commented 6 years ago

First draft for OTFM an OTFI file format schemas: https://github.com/otland/forgottenserver/commit/d01e041409ab3877c5c71fd066bfa7577b73249a

Some open design questions:

conde2 commented 6 years ago

should sectors have explicit sizes? I think yes, it makes loading more consistent, faster (no need to calculate sectors) and I can't see drawbacks on this.

can we make lazy loading sectors easier? Easier in which way ?

is it feasible to use the Item schema to store items in the database? How it will work with custom attributes ?

ranisalt commented 6 years ago

I think you are working too much on porting the current format rather than designing a new format from the ground up, e.g.

FB has the advantage of being pluggable should we need any other info in the future, and itself would reduce file size by a lot.

Also, new map format and new item format should be designed separately.