search

datacamp / pythonwhat

Verify Python code submissions and auto-generate meaningful feedback messages.
http://pythonwhat.readthedocs.io/
GNU Affero General Public License v3.0
69 stars 31 forks source link

Speed things up? has_equal_x: make copy=False the default? #286

Closed filipsch closed 5 years ago

filipsch commented 6 years ago

Every time an SCT uses has_equal_value(), has_equal_output() or has_equal_error(), by default it sets copy = True, which means that all variables that are in the environment are copied over to a new environment in which the actual focused expression is executed. This was done to make sure there were no nasty side effects of running expressions that were focused on, but from an efficiency standpoint is pretty terrible. I wonder what would happen if we'd make copy=False the default, and somehow figure out which exercises need the explicit copy=True value set. The performance benefits could be huge, but we should proceed with care. FYI, There are 64 exercises that use copy=False.

@machow do you have any thoughts on this?

machow commented 6 years ago

This one seems tricky, since I think issues that pop up will come from running submissions that are different from the solution code--so won't be picked up by the validator.

On the other hand, it seems possible that setting copy = True was just way too defensive, and the concern I just mentioned is purely theoretical :p

One way to check might be to make a branch for a course that runs a lot of these SCTs, and then kick the tires on some of the exercises. I'm happy to spend 30 mins or so trying different submissions

hermansje commented 5 years ago

The automatic disabling of copying the environment when only accessing a value has been improved: https://github.com/datacamp/pythonwhat/pull/367/commits/5761f44c4a988f9538fc03f851583da261223dd5