Asset IDs Are Different for Same Path on Web and Desktop

[zicklag]

Bevy version

v0.8.1

What you did

Run this code, on web and desktop builds.

let path = "map/levels/lev01.map.json";
let asset_path: AssetPath = path.into();
let handle = Handle::<MapMeta>::weak(asset_path.clone().into());
info!(?path, ?asset_path, ?handle, "Level 1 handle");

Observe the numbers for the handle in the logged output.

What went wrong

Expected: The handle's generated on web to be the same as the handle generated on native.

Actual result: The handle IDs are different on web and desktop

Additional information

I wanted to send the asset handles across the network for my networked game, but I'm going to have a Bevy server running on native, and clients running on web, so the difference in the handle ID is a blocker for that.

Having known that the Handle ID was a hash of the asset path, I was rather surprised to find that they were different on the different platforms.

I suspect that this has something to do with AHash using AES-NI instructions on native for extra prerformance, and then falling back to a different implementation on web, where those instructions don't exist.

Is This a Bug?

I'd say it's debatable whether or not this is really a but and needs to be fixed. I would definitely like it fixed for my use-case, and I imagine other people would like to do the same thing, but I'd be good to collect thoughts from other users/developers.

Solution

The only solution I think is to change the hash algorithm. Specifically mentioned in the AHash readme:

Specifically, aHash is not intended for network use or in applications which persist hashed values. (In these cases HighwayHash would be a better choice)

It sounds like we should look into HighwayHash.

Feature Flag

Since AHash's algorithm is specifically designed for speed only, it could have speed advantages over HighwayHash, so there might still be users who'd want to use it instead.

Maybe we could default to HighwayHash and provide a feature flag for using AHash instead?

I think we'd only want to do that, though, if we could find benchmarking or other evidence that shows AHash actually being faster in Bevy.

[mockersf]

Could you send the actual path over the network, or keep your own hash?

[zicklag]

I could, but I have nested structures for my custom asset that store handles to link to their related assets, so I just have a lot of handles, and storing the path and the handle felt redundant, when the handle can be derived from the hash ( except for the hash portability issue ).

If we think this doesn't need to be changed, then I'll probably just create a custom type that derefs to a handle, and also stores an Arc<str> for the original asset path, so that I can use the Arc<str> for serialization over the network. That probably wouldn't be that bad.

[zicklag]

While we are on the topic of different hashers, it was brought up on the Rust forum while I was asking a question about hash collision probabilities, that 64 bits might not be enough to prevent hash collisions, though that's a different discussion, it is relevant if we are changing the hashing algorithm.

I just don't know how to really tell whether 64 bits is sufficient or not. I don't know enough about the topic.

---

On the topic of which hashing algorithm to use, though, 64 bit question aside, from the HighwayHash readme:

I think for bevy games, we are usually going to be sitting well under the 64 byte length as far as asset path length goes.

According to this benchmark anyway, it looks like FxHash is the fastest, at the cost of not being resistant to DDOS, but Bevy intentionally makes it's AHash predictable and un-resistant to DDOS, so that is not a disadvantage to us at all.

If we used FxHash, we might get a performance increase, but I'm not sure whether it's platform independent or not yet.

[zicklag]

I ended up using an AssetHandle struct that is serializable and can be sent over the network by holding the asset path along with the asset handle: https://gist.github.com/zicklag/5314dc3ffabeff575939e3e0185b083f.

It's a little more annoying to use because you've got an extra wrapping around the Handle<T> that most bevy components need, but it's not too bad, and the serialization works great.

[djeedai]

I think for bevy games, we are usually going to be sitting well under the 64 byte length as far as asset path length goes.

Maybe I'm misunderstanding that sentence, but are you saying you expect asset paths on disk to be less than 64 bytes always? Because that's a severely low limit, even being relative to the asset folder. That may work for smaller games will low asset count, but as soon as you do something remotely large and complex and want to organize in sub-folders, you're going to hit that 64 bytes limit pretty fast. Also think about languages that may include non-ASCII characters in their paths (not sure if Bevy supports it, but probably should), which use multi-byte encoding so will eat up that quota even faster.

That being said, I agree that FxHash looks good for smaller payload, and is still reasonable up to 256 bytes, and even above 10GB/s up to 512 bytes, so the conclusions are probably unaffected.

[djeedai]

@zicklag where's the graph from? I can't find it on https://github.com/google/highwayhash/blob/master/README.md, and I can't find any reference of FxHash either. Thanks!

[zicklag]

I got the graph from the highway-rs README.

Maybe I'm misunderstanding that sentence, but are you saying you expect asset paths on disk to be less than 64 bytes always? Because that's a severely low limit, even being relative to the asset folder.

I was thinking that we'd probably be under 64 bytes in not all cases, but most. But yeah, you're probably right that once you get to a certain level, that's really small.