Closed zhouyan closed 5 years ago
Hi,
No this is not intentional. However, if you have an fbs file, why not generate the Julia from it as well instead of writing it by hand? The whole point of the 0.5 release is to support this.
You can find a fork of Flatbuffers with Julia in flatc here.
Unless there is a compelling reason why you can't use the Julia flatc backend in this case the behaviour probably won't be changed back.
First, thanks for the effort of developing flatc support. It is really great.
But there are a few reasons not to use the flatc at least for now,
If it is not too difficult, maintaining backward compatibility would be great. It allows people to migrate gradually instead of risking breaking critical code. For example, at the moment I have no choice but to pin it to v0.4.0, even in testing environment. However, it is much preferable to be able to upgrade, and migrate little by little. Otherwise it leaves people two choices:
In the end, it is more of a chicken egg problem. If I can upgrade the package first, then upgrading code base using flatc
gradually, it makes the upgrading much more desirable.
Cool, thanks for the explanation, and I can certainly understand your frustration. There is a PR open on the upstream Flatbuffers project, which I'm hoping can be merged in the coming weeks, so hopefully you shouldn't have to wait too long on this.
The motivation for this change is that a lack of a string field is different from an empty string, and the new way is more in line with the existing Flatbuffers code. This helps with getting the PR through :)
I'm not aware of any current breakages in Feather.jl - can you elaborate on this? I'm happy to update Feather.jl for compatibility.
How difficult would it be to run the experiment of upgrading everything and seeing what breaks? This would be valuable since I don't have a codebase of that size to work with. Feel free to post any problems here and I'll do my best to help.
Otherwise, yes unfortunately until it's merged the best thing to do would be to pin the package and wait.
For feather the issue is with Timestamp type metadata, which has a time zone file which maybe empty and missing in the feather file written by other languages (tried Python and C++). Though feather written by a Julia itself can be read back properly.
Thanks a lot if it can be updated.
It will of course be great to have the PR for upstream merged soon. But it does not solve all problems yet. By switching to flatc
it means a lot types need to be changed. Some may have a different name than it has been defined now. That is the most difficult part to update. A simple example, one of the flatbuffers type we have has a field named begin
, and in Julia the corresponding field is named _begin
. I guess the flatc generated code will have similar convention to avoid conflict with keywords. But it may or may not be the same as the convention we use now, this means a lot places where the generated type is used will need code to be updated to use new field name. Unlike say C++, one can make the change and just fix all compile errors, in a dynamic language like a Julia, such changes may introduce errors not detected until runtime. Of course code coverage and unit tests help, but it is difficult to provide 100% coverage.
It is small cases like this making upgrading risky and needing careful planing and thus why backward compatibility is invaluable
(Of course in the specific example above, it may seems silly to have a field named begin
to begin with. But remember that flatbuffer is a cross language format, the schema has been used with C++ and Python long before we started adding Julia support, and changing schema just for one language is not an option unless I want everyone to start shouting at me (e.g., the guy responsible for the Python codebase). And it is difficult to avoid all keys from all languages)
In the more specific case of string, on one hand I agree a missing string is not the same as an empty string. On the hand, within the context of flatbuffer I think it does default to empty string, or at least this seems to be how it is handled in other languages. Another way to see it is that every field need to be initialized by a default value if not presented in the buffer, so they need a sensible default unless specified otherwise. For string, the sensible one I believe is empty string.
Sent from my iPad
On Dec 31, 2018, at 05:14, Rowan notifications@github.com wrote:
Cool, thanks for the explanation, and I can certainly understand your frustration. There is a PR open on the upstream Flatbuffers project, which I'm hoping can be merged in the coming weeks, so hopefully you shouldn't have to wait too long on this.
The motivation for this change is that a lack of a string field is different from an empty string, and the new way is more in line with the existing Flatbuffers code. This helps with getting the PR through :)
I'm not aware of any current breakages in Feather.jl - can you elaborate on this? I'm happy to update Feather.jl for compatibility.
How difficult would it be to run the experiment of upgrading everything and seeing what breaks? This would be valuable since I don't have a codebase of that size to work with. Feel free to post any problems here and I'll do my best to help.
Otherwise, yes unfortunately until it's merged the best thing to do would be to pin the package and wait.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
Leave it with me. The use case of combining flatc generated code with handwritten Julia is certainly one for which we don't have any tests, hence the unintentional breakage. Seems like there is a similar lack of cross-language tests in Feather.jl since we are not failing tests there either.
It may be a trivial change to move back to the old behaviour, or there may be other complications. I'm away on holiday at the moment but I'll investigate in the new year and post an update here.
Great, thanks a lot
Sent from my iPad
On Dec 31, 2018, at 10:13, Rowan notifications@github.com wrote:
Leave it with me. The use case of combining flatc generated code with handwritten Julia is certainly one for which we don't have any tests, hence the unintentional breakage. Seems like there is a similar lack of cross-language tests in Feather.jl since we are not failing tests there either.
It may be a trivial change to move back to the old behaviour, or there may be other complications. I'm away on holiday at the moment but I'll investigate in the new year and post an update here.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
Just released 0.5.2, which reverts back to the old behaviour. Hopefully this fixes your issue :)
Thanks a lot. I have just come back from holiday and tested the new version, all things to work fine now and I will start testing the flatc
support on development branches and keep you updated if any issues happens.
I encountered the problem when using Feather.jl after upgraded FlatBuffers.jl. Below is a minimal example that reproduce the issue.
Below is the
fbs
file,Below is a simple python script that use generated python module to generate a binary file containing the flat buffer bytes (this can be done in Julia but here I am using python just to rule out possibility that the buffer is corrupted itself. In the place where I encountered the issue first it was generated by C++),
Below is the Julia test file,
Blow is the commands
When using
v0.4.0
, it gives the expected result,But after upgrading, it gives the error,
A workaround would be to define a constructor for
Foo
that acceptNothing
. However, for types with many fields, this fast become infeasible. Besides, this change breaks too much existing codebase.Is this an intentional break? If yes, would you consider to change it back?