Visible Import Types retrieval

[elatoskinas]

Visible Type Extraction

Continuation of https://github.com/mir-am/ML4SE/pull/3 Resolves: https://github.com/mir-am/ML4SE/issues/2

Adds a file that allows to do the following:

• Retrieve a list of all import statements of a file (including transitive imports)
• Retrieve a list of all (resolved) modules for the given file that it imports
• Retrieve a list of visible types that originate from imports for the given file

Changes from original PR

• Import types now get resolved to their qualified names. So if we have a type originating from scikit.linalg, the type name will then now be "scikit.linalg"
• Imports are now checked recursively. We check imports of imports
• Refactoring changes to make it easier to use
• Minor bug fixes (mainly addressing false negatives for types in some cases)

Usage & Integration

The type extractor is not yet integrated into the pipeline. Integration is straight forward:

1. Suppose we have an input file (path) file
2. Create a new ModuleExtractor
3. Call ModuleExtractor.getTypes(file)
4. The returned result is a set of types. We can then keep it as a set, or convert it to a list. This is our final result output.

[elatoskinas]

@mir-am The implemented extractor is in module_type_extractor.py. All other files are there solely for testing purposes, and we will remove them once we finalize this PR.

[mir-am]

@elatoskinas Thanks for the work. Do you think that we can run the import tool for 5,000 Python projects? Considering the issues we discussed so far. Moreover, I will train the model of TypeWriter with the visible types that you sent.

[elatoskinas]

@mir-am Yes, we can certainly train it on 5000 repos; I'll get to this soon!

[elatoskinas]

Closing, moved to repo: https://github.com/saltudelft/visible-type-extractor