How to import user-defined module correctly?

[tjuwyh]

Hi Alasdair,

I want to experiment with some new modules, but I cannot figure out that how to import classes except by adding "--include-package" in the command line. I'm wondering how do the modules defined by you imported and what can I do to import my new modules correctly?

Thanks very much!

[alasdairtran]

If you're adding the custom module directly in my package, then you don't need to use "--include-package". You just need to import them in the __init__.py file. Have a look at tell/models/__init__.py and tell/modules/__init__.py, for example, to see how I exposed my custom modules.

[tjuwyh]

Thanks! I followed your advice and it works. And I'm still wondering how are the modules correctly imported. The AllenNLP advice to use "--include-package" or write a .allennlp_plugins but I cannot find any .allennlp_plugins file. So would you please share with me some details on how you made it?

Besides, I found it is feasible to import different modules with the same name in the __init__.py file. For example, If I import two 'TransformerFlattenedModel' in tell/models/__init__.py from .transformer_flattened_modified import TransformerFlattenedModel from .transformer_flattened import TransformerFlattenedModel and register the TransformerFlattenedModel classes as different types: @Decoder.register('dynamic_conv_decoder_flattened_modified') and @Decoder.register('dynamic_conv_decoder_flattened') Things turn out that the trainer can still use the correct class. Is it implemented by AllenNLP or the provided tell environment?

[alasdairtran]

Ah if we're using the allennlp train command, then we would indeed need to have --include-package. But because we're using tell train (which is my package), then all the modules in the tell package would be available automatically (as long as you put them in __init__.py because Python automatically reads these files). When I was doing this project, mixed precision wasn't supported in AllenNLP and I wanted to use yaml instead of json config files, so that's why I duplicated AllenNLP's training code and made some small modifications.

About the duplicated class name, that's an interesting observation! I didn't know that it would still work. I guess the name given to the registry is what matters. And the class name is irrelevant (although it's still better practice to give different classes distinct names). The way the registry works is inherited from AllenNLP. You can see, for e.g., that Decoder is a sublass of allennlp.common.registrable.Registrable. I haven't looked too carefully into how Registrable works myself.

[tjuwyh]

Thanks for your help, it is very useful for me!