Xu_PyProbaTyping

[ctreude]

The idea of using aspects such as attribute access, data flow, variable names, and explicit type checks to infer Python types is novel and innovative.

The corresponding artifact provided here is insightful, useful and usable. The tool works as advertised and I did not encounter any problems following the instructions using VMWare Workstation. The artifact contents all make sense and I appreciate that the artifact contains all relevant benchmark data to reproduce the results from the corresponding FSE paper.

Typo in the pdf: "fo most variables"

I'm happy to recommend the inclusion of this dataset in the online repository, the social media campaign and the award session.

[hongyujohn]

[Summary] This artifact package includes the implementation of a probabilistic type inference engine for Python. The engine is capable of inferring types using a variety of "type hints". I am able to run the tool and see the results. I recommend acceptance.

[Insightful] Inferring types for dynamic languages is challenging. The proposed approach uses a feasible combination of different type hints to optimize the type inference. The artifact should be more convincing if the presented tool is compared to machine-learning based approaches.

[Useful] The tool could be practically useful. Several benefits have been proposed, including avoiding type errors and getting hints in API usage.

[Usable] The authors made the artifact available with a VMware image, which mitigated the complexity of download and installation. Accompanied with a well-structured and well-presented README file, the tool is generally easy to use. By following the instructions step by step, I was able to run the inference and produce results. However, I was a little confused about the time efficiency of running the artifact. While it is stated in the paper that typing a variable takes less than 1 second, inferring a single project (say bs4) in my reproduction took over 15 minutes. Perhaps this is because of a large number of variables. A clearer explanation on the time cost of the inference can make the tool more user friendly. Also, it would be nicer if some suggestions can be offered to help users set appropriate parameter values.

[Felienne]

This paper proposes a tool to do type inference for Python. The novel approach here is that they use a probabilistic approach. The author report over 80% of precision, which seems impressive!

The tool is well documented, and using the steps in the paper I managed to run the tool. Like @hongyujohn the performance was an issue, although I have to admit I ran it on an old MacBook.

I am still in favor of inclusion of this artefact though.

[timm]

Note these labels are still "under discussion" and are still subject to change prior to the final notifications Friday.