Development andres - Githubissues

andreshyer commented 4 years ago

This is fairly large PR. I strongly encourage for those of you who review to go into depth reviewing this PR.

Added features

QsarDB databases can now be directly imported into Neo4j
Scaled data has been added for: predictions.csv, RMSE, MSE, and R2 (averages and stds)
When model.analyze() is run, pretty histograms of pva and scaled_pva is generated
predictions.csv and scaled_predictions.csv now has a column ['pred_error']. This columns is simply abs(['actual'] - ['pred_avg'])

Ontology changes

Removed NotTuned node
Removed (:Algorithm)-[]->(:NotTuned) relationship
Added (:MLModel)-[:USES_TUNING]->(:Tuning)
Made it so Models that do not use tuning have no tuning node

Bug Fixes

No longer forced to insert all data into MySql when using MySql to featurize data

Other changes

One base function to import models, output directories, and QsarDB databases.
Neo4j files have all been condesed into models_to_neo4j.py for changeability and readability
This speed up insertion even faster. All regression datasets (including logP14k) can be inserted into Neo4j in roughly 2 minutes

pep8speaks commented 4 years ago

Hello @andreshyer! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file core/analysis.py:

Line 156:80: E501 line too long (83 > 79 characters) Line 157:80: E501 line too long (85 > 79 characters) Line 165:80: E501 line too long (109 > 79 characters)

In the file core/features.py:

Line 15:80: E501 line too long (116 > 79 characters) Line 16:80: E501 line too long (114 > 79 characters) Line 17:80: E501 line too long (118 > 79 characters) Line 21:80: E501 line too long (105 > 79 characters)

In the file core/ingest.py:

Line 28:80: E501 line too long (90 > 79 characters)

In the file core/models.py:

Line 131:80: E501 line too long (92 > 79 characters) Line 137:80: E501 line too long (107 > 79 characters) Line 141:80: E501 line too long (87 > 79 characters)

In the file core/neo4j/__init__.py:

Line 2:1: W391 blank line at end of file

In the file core/neo4j/models_to_neo4j.py:

Line 21:80: E501 line too long (96 > 79 characters) Line 22:80: E501 line too long (86 > 79 characters) Line 26:80: E501 line too long (112 > 79 characters) Line 27:80: E501 line too long (110 > 79 characters) Line 28:80: E501 line too long (111 > 79 characters) Line 29:80: E501 line too long (94 > 79 characters) Line 31:80: E501 line too long (81 > 79 characters) Line 33:80: E501 line too long (102 > 79 characters) Line 35:80: E501 line too long (85 > 79 characters) Line 36:80: E501 line too long (116 > 79 characters) Line 37:80: E501 line too long (100 > 79 characters) Line 45:80: E501 line too long (92 > 79 characters) Line 46:80: E501 line too long (116 > 79 characters) Line 55:80: E501 line too long (119 > 79 characters) Line 78:80: E501 line too long (83 > 79 characters) Line 86:80: E501 line too long (80 > 79 characters) Line 94:80: E501 line too long (80 > 79 characters) Line 95:80: E501 line too long (82 > 79 characters) Line 97:80: E501 line too long (96 > 79 characters) Line 108:80: E501 line too long (116 > 79 characters) Line 117:80: E501 line too long (112 > 79 characters) Line 118:80: E501 line too long (85 > 79 characters) Line 123:80: E501 line too long (102 > 79 characters) Line 126:80: E501 line too long (84 > 79 characters) Line 127:80: E501 line too long (92 > 79 characters) Line 128:80: E501 line too long (83 > 79 characters) Line 132:80: E501 line too long (97 > 79 characters) Line 138:80: E501 line too long (109 > 79 characters) Line 139:80: E501 line too long (112 > 79 characters) Line 140:80: E501 line too long (113 > 79 characters) Line 154:80: E501 line too long (94 > 79 characters) Line 158:80: E501 line too long (90 > 79 characters) Line 162:80: E501 line too long (83 > 79 characters) Line 171:80: E501 line too long (89 > 79 characters) Line 179:80: E501 line too long (114 > 79 characters) Line 180:80: E501 line too long (100 > 79 characters) Line 192:80: E501 line too long (112 > 79 characters) Line 195:80: E501 line too long (116 > 79 characters) Line 196:80: E501 line too long (81 > 79 characters) Line 201:80: E501 line too long (85 > 79 characters) Line 242:80: E501 line too long (99 > 79 characters) Line 243:80: E501 line too long (94 > 79 characters) Line 252:80: E501 line too long (99 > 79 characters) Line 255:80: E501 line too long (82 > 79 characters) Line 257:80: E501 line too long (92 > 79 characters) Line 263:80: E501 line too long (91 > 79 characters) Line 267:80: E501 line too long (110 > 79 characters) Line 268:80: E501 line too long (110 > 79 characters) Line 275:80: E501 line too long (115 > 79 characters) Line 276:80: E501 line too long (115 > 79 characters) Line 297:80: E501 line too long (86 > 79 characters) Line 298:80: E501 line too long (88 > 79 characters) Line 299:80: E501 line too long (88 > 79 characters) Line 300:80: E501 line too long (88 > 79 characters) Line 335:80: E501 line too long (81 > 79 characters) Line 370:80: E501 line too long (112 > 79 characters) Line 371:80: E501 line too long (120 > 79 characters) Line 383:1: W293 blank line contains whitespace Line 385:80: E501 line too long (105 > 79 characters) Line 389:80: E501 line too long (107 > 79 characters) Line 389:108: W291 trailing whitespace Line 390:80: E501 line too long (104 > 79 characters) Line 392:1: W293 blank line contains whitespace Line 394:80: E501 line too long (115 > 79 characters) Line 395:80: E501 line too long (80 > 79 characters) Line 398:1: W293 blank line contains whitespace Line 399:80: E501 line too long (80 > 79 characters) Line 402:1: W293 blank line contains whitespace Line 403:80: E501 line too long (83 > 79 characters) Line 406:1: W293 blank line contains whitespace Line 410:1: W293 blank line contains whitespace Line 412:80: E501 line too long (103 > 79 characters) Line 414:80: E501 line too long (81 > 79 characters) Line 415:80: E501 line too long (97 > 79 characters) Line 419:80: E501 line too long (104 > 79 characters) Line 433:1: W293 blank line contains whitespace Line 435:1: W293 blank line contains whitespace Line 436:80: E501 line too long (96 > 79 characters) Line 437:80: E501 line too long (83 > 79 characters) Line 439:1: W293 blank line contains whitespace Line 441:80: E501 line too long (85 > 79 characters) Line 442:80: E501 line too long (85 > 79 characters) Line 450:80: E501 line too long (111 > 79 characters) Line 451:80: E501 line too long (104 > 79 characters) Line 459:1: W293 blank line contains whitespace Line 460:58: W291 trailing whitespace Line 463:80: E501 line too long (87 > 79 characters) Line 463:88: W291 trailing whitespace Line 464:80: E501 line too long (95 > 79 characters) Line 465:62: W291 trailing whitespace Line 466:1: W293 blank line contains whitespace Line 486:1: W293 blank line contains whitespace Line 496:1: W293 blank line contains whitespace Line 506:80: E501 line too long (118 > 79 characters) Line 517:57: W291 trailing whitespace Line 526:80: E501 line too long (118 > 79 characters) Line 527:80: E501 line too long (115 > 79 characters) Line 539:80: E501 line too long (95 > 79 characters) Line 542:80: E501 line too long (91 > 79 characters) Line 546:80: E501 line too long (115 > 79 characters) Line 550:80: E501 line too long (111 > 79 characters) Line 551:80: E501 line too long (111 > 79 characters) Line 552:80: E501 line too long (112 > 79 characters) Line 579:80: E501 line too long (91 > 79 characters) Line 580:80: E501 line too long (84 > 79 characters) Line 581:80: E501 line too long (86 > 79 characters) Line 582:80: E501 line too long (88 > 79 characters) Line 591:80: E501 line too long (86 > 79 characters) Line 599:80: E501 line too long (83 > 79 characters) Line 600:80: E501 line too long (81 > 79 characters) Line 603:80: E501 line too long (83 > 79 characters) Line 608:80: E501 line too long (103 > 79 characters) Line 613:80: E501 line too long (108 > 79 characters) Line 620:80: E501 line too long (89 > 79 characters) Line 621:1: W293 blank line contains whitespace Line 623:80: E501 line too long (115 > 79 characters) Line 623:116: W291 trailing whitespace Line 624:80: E501 line too long (117 > 79 characters) Line 624:118: W291 trailing whitespace Line 626:1: W293 blank line contains whitespace Line 627:80: E501 line too long (113 > 79 characters) Line 628:80: E501 line too long (115 > 79 characters) Line 629:80: E501 line too long (116 > 79 characters) Line 634:80: E501 line too long (83 > 79 characters) Line 635:80: E501 line too long (85 > 79 characters) Line 636:80: E501 line too long (88 > 79 characters) Line 638:80: E501 line too long (82 > 79 characters) Line 640:80: E501 line too long (102 > 79 characters) Line 641:80: E501 line too long (90 > 79 characters) Line 642:80: E501 line too long (95 > 79 characters) Line 643:1: W293 blank line contains whitespace Line 645:80: E501 line too long (96 > 79 characters) Line 646:80: E501 line too long (107 > 79 characters) Line 648:80: E501 line too long (104 > 79 characters) Line 651:80: E501 line too long (100 > 79 characters) Line 652:80: E501 line too long (102 > 79 characters) Line 653:80: E501 line too long (104 > 79 characters) Line 660:80: E501 line too long (111 > 79 characters) Line 661:80: E501 line too long (111 > 79 characters) Line 669:80: E501 line too long (83 > 79 characters) Line 689:1: W293 blank line contains whitespace Line 695:1: W293 blank line contains whitespace Line 697:80: E501 line too long (85 > 79 characters) Line 703:80: E501 line too long (88 > 79 characters) Line 704:80: E501 line too long (120 > 79 characters) Line 705:80: E501 line too long (126 > 79 characters) Line 706:80: E501 line too long (116 > 79 characters) Line 707:80: E501 line too long (125 > 79 characters) Line 713:80: E501 line too long (111 > 79 characters) Line 714:80: E501 line too long (94 > 79 characters) Line 721:80: E501 line too long (95 > 79 characters) Line 729:1: W293 blank line contains whitespace Line 744:80: E501 line too long (113 > 79 characters) Line 753:80: E501 line too long (118 > 79 characters) Line 769:1: W293 blank line contains whitespace Line 774:80: E501 line too long (84 > 79 characters)

In the file core/storage/mysql.py:

Line 45:80: E501 line too long (87 > 79 characters) Line 46:80: E501 line too long (99 > 79 characters) Line 172:80: E501 line too long (85 > 79 characters) Line 177:80: E501 line too long (88 > 79 characters) Line 179:80: E501 line too long (108 > 79 characters) Line 181:80: E501 line too long (103 > 79 characters) Line 201:80: E501 line too long (83 > 79 characters) Line 246:80: E501 line too long (107 > 79 characters) Line 247:80: E501 line too long (84 > 79 characters)

In the file core/storage/qsar_export.py:

Line 110:80: E501 line too long (85 > 79 characters) Line 158:80: E501 line too long (107 > 79 characters) Line 190:80: E501 line too long (80 > 79 characters)

In the file core/storage/qsar_to_neo4j.py:

Line 18:80: E501 line too long (99 > 79 characters) Line 19:80: E501 line too long (86 > 79 characters) Line 60:80: E501 line too long (81 > 79 characters) Line 102:80: E501 line too long (91 > 79 characters) Line 155:80: E501 line too long (84 > 79 characters) Line 162:80: E501 line too long (94 > 79 characters) Line 170:80: E501 line too long (118 > 79 characters) Line 171:80: E501 line too long (104 > 79 characters) Line 172:80: E501 line too long (102 > 79 characters) Line 172:95: E711 comparison to None should be 'if cond is not None:' Line 173:80: E501 line too long (97 > 79 characters) Line 184:80: E501 line too long (96 > 79 characters) Line 186:80: E501 line too long (101 > 79 characters) Line 186:94: E711 comparison to None should be 'if cond is not None:' Line 193:80: E501 line too long (82 > 79 characters) Line 213:80: E501 line too long (110 > 79 characters) Line 250:80: E501 line too long (100 > 79 characters) Line 252:80: E501 line too long (83 > 79 characters) Line 271:80: E501 line too long (80 > 79 characters) Line 274:80: E501 line too long (86 > 79 characters) Line 295:80: E501 line too long (103 > 79 characters) Line 296:80: E501 line too long (80 > 79 characters) Line 297:80: E501 line too long (81 > 79 characters) Line 302:80: E501 line too long (84 > 79 characters) Line 304:80: E501 line too long (85 > 79 characters) Line 305:80: E501 line too long (90 > 79 characters) Line 308:80: E501 line too long (92 > 79 characters) Line 309:80: E501 line too long (103 > 79 characters) Line 310:80: E501 line too long (113 > 79 characters) Line 315:80: E501 line too long (81 > 79 characters) Line 329:80: E501 line too long (84 > 79 characters) Line 336:80: E501 line too long (97 > 79 characters)

In the file core/train.py:

Line 67:80: E501 line too long (93 > 79 characters) Line 70:80: E501 line too long (80 > 79 characters) Line 78:80: E501 line too long (95 > 79 characters) Line 79:80: E501 line too long (106 > 79 characters) Line 85:80: E501 line too long (106 > 79 characters) Line 89:80: E501 line too long (81 > 79 characters)

In the file main.py:

Line 71:80: E501 line too long (150 > 79 characters) Line 72:80: E501 line too long (80 > 79 characters) Line 75:80: E501 line too long (119 > 79 characters) Line 80:80: E501 line too long (111 > 79 characters) Line 88:80: E501 line too long (81 > 79 characters) Line 90:80: E501 line too long (101 > 79 characters) Line 101:80: E501 line too long (88 > 79 characters) Line 123:80: E501 line too long (101 > 79 characters) Line 125:80: E501 line too long (102 > 79 characters) Line 147:80: E501 line too long (80 > 79 characters) Line 152:80: E501 line too long (90 > 79 characters) Line 161:80: E501 line too long (114 > 79 characters) Line 213:80: E501 line too long (102 > 79 characters)

Comment last updated at 2020-09-04 01:58:07 UTC

andreshyer commented 4 years ago

@qle2 Hey Quang, you point out some great stuff. You noted that you did not see fragments being merged. Is this still a problem? Because I am unable to replicate that on my machine. Also, did you notice any big underlying issues? Such as missing nodes or relationships?

andreshyer commented 4 years ago

@qle2 All major issues have been addressed. Only thing left to address from your comments is the optimization using yield to not kill computers

andreshyer commented 4 years ago

uncertainity, prediction_average, and predition_average_error has been added Screenshot from 2020-09-03 13-09-28

andreshyer commented 4 years ago

Histograms have been prettied up Aw020_20200903-140004_hist_Predicted Aw020_20200903-140004_hist_Actual

Dlux804 / McQuade-Chem-ML

Development andres #87

Comment last updated at 2020-09-04 01:58:07 UTC