Closed epetros closed 4 years ago
@epetros oops, it's been fixed in the latest my bad https://github.com/lucidrains/routing-transformer/commit/f6ce3bbcd07add17e9bb0be281850ee8a6045646 how is Routing Transformer working for you? well??
Thank you, but I still face the same issue, works only with v0.8.4. Am using enwik8_deepspeed with my own dataset, seems to be very fast and loss is decreasing but guess it's still early (loss ~0.9) to get nice generation results.
@epetros fixed for real now lol https://github.com/lucidrains/routing-transformer/commit/1341f6987028f5bebca009ee4925baffd0572653
It's fixed, thanks.
Greetings, Previously was able to save and load checkpoints, but today I get: RuntimeError: Error(s) in loading state_dict for AutoregressiveWrapper: Missing key(s) in state_dict: "net.net.routing_transformer.layers.blocks.0.f.net.fn.local_attn.rel_pos.weights", "net.net.routing_transformer.layers.blocks.1.f.net.fn.local_attn.rel_pos.weights", "net.net.routing_transformer.layers.blocks.2.f.net.fn.local_attn.rel_pos.weights", "net.net.routing_transformer.layers.blocks.3.f.net.fn.local_attn.rel_pos.weights", "net.net.routing_transformer.layers.blocks.4.f.net.fn.local_attn.rel_pos.weights", "net.net.routing_transformer.layers.blocks.5.f.net.fn.local_attn.rel_pos.weights", "net.net.routing_transformer.layers.blocks.6.f.net.fn.local_attn.rel_pos.weights", "net.net.routing_transformer.layers.blocks.7.f.net.fn.local_attn.rel_pos.weights".
Help please, Thanks