search

saltudelft / dl-type-python

Deep Learning-based type inference for Python
GNU General Public License v3.0
7 stars 4 forks source link

Pipeline Exception Catching for Extractor #3

Closed elatoskinas closed 4 years ago

elatoskinas commented 4 years ago

Currently, the entire project will fail processing if one file fails to process in the pipeline. To extract more data, we should consider catching exceptions on the file, and let the project parsing continue.

For instance, some projects might have artifact files that cause the extractor to fail, but are otherwise projects that are completely valid apart from those artifact files.

The small fix will catch these exceptions and skip the files. It can be improved upon in the future from the code side, but will do for now.

mir-am commented 4 years ago

Currently, the entire project will fail processing if one file fails to process in the pipeline. To extract more data, we should consider catching exceptions on the file, and let the project parsing continue.

For instance, some projects might have artifact files that cause the extractor to fail, but are otherwise projects that are completely valid apart from those artifact files.

The small fix will catch these exceptions and skip the files. It can be improved upon in the future from the code side, but will do for now.

Thanks for the fix. I will test it by the end of the week. It should now extract more projects than the previous one without the fix.

mir-am commented 4 years ago

I will merge the PR and close it.