Apply black to Tests/ - Githubissues

peterjc commented 4 years ago

This is a spin out from #2008, where we have agreed to adopt the black code formatting tool for Biopython:

That issue focused on applying black to the main code base (Bio/, BioSQL/, and also Scripts/ and Doc/examples/).

This issue is for applying black to the Biopython test suite.

In principle this is a good issue for new contributors. However, many of the tests include expected data values, so compared to the Biopython main code there are more likely to be cases where black does a bad job - which will need some discussion (e.g. can it be improved with extra brackets, or do we turn off black for a portion of a test file?).

You will need to install black, flake8, and associated plugins as described in https://github.com/biopython/biopython/blob/master/CONTRIBUTING.rst - and we recommend turning on the git pre-commit hook).

Example workflow:

git checkout master # and make sure it is up to date
git checkout -b black_test_XXX # start a new branch
black Tests/test_XXX.py  # do the conversion
flake8 Tests/test_XXX.py # check for string concatenation etc
emacs Tests/test_XXX.py  # review and make any fixes
git commit Tests/test_XXX.py -m "Apply black to Tests/test_XXX.py"
git push my_fork black_text_XXX  # push the branch to your personal GitHub

To avoid duplication of effort, please comment here if you are going to work on a particular file, or group of files.

svalqui commented 4 years ago

I'll work on test_PDB*

svalqui commented 4 years ago

I'll work on test_SeqIO*

svalqui commented 4 years ago

I'll work on test_SearchIO* files.

svalqui commented 4 years ago

I'll work on test_pro* files.

svalqui commented 4 years ago

I'll work on test_Phylo* files.

svalqui commented 4 years ago

I'll work on test_PAML* files.

svalqui commented 4 years ago

I'll work on test_T* files.

svalqui commented 4 years ago

I'll skip test_Tutorial.py as is raising errors with some doctest on the tutorial. ValueError: 4 Tutorial doctests failed: test_chapter_entrez_line_00202, test_chapter_entrez_line_00325, test_chapter_entrez_line_00419, test_chapter_entrez_line_00466,

svalqui commented 4 years ago

I'll work on SCOP files

peterjc commented 4 years ago

You can try python run_tests.py test_Tutoturial.py --offline but it sounds like four of the online Entrez tests in the Tutorial were failing. Please confirm if that was only after running black or not. If not, please open a new issue. Thanks.

svalqui commented 4 years ago

It was before running black, I have logged an issue for that.

svalqui commented 4 years ago

I'll work on test_PopGen* files

svalqui commented 4 years ago

I'll work on test_phenotype* files.

svalqui commented 4 years ago

I'll work on test_NCBI* files.

svalqui commented 4 years ago

test_NCBI_qblast.py is failing, before and after black, so not black related, I'll apply black style and then log an issue.

svalqui commented 4 years ago

I'll work on test_m* files.

svalqui commented 4 years ago

I'll work on test_l* files.

svalqui commented 4 years ago

I'll work on test_k* files.

svalqui commented 4 years ago

I'll work on test_HMM* files.

svalqui commented 4 years ago

I'll work on test_g* files.

svalqui commented 4 years ago

I'll work on test_Entrez files.

svalqui commented 4 years ago

I'll work on test_Emboss files.

svalqui commented 4 years ago

I'll work on test_BioSQL files.

svalqui commented 4 years ago

I'll work on test_AlignIO files.

svalqui commented 4 years ago

All that could had been grouped had black style applied, now looking at individual files.

gandalfthegrape commented 4 years ago

Hello, I'm still new to contributing. Unless I'm mistaken, it looks like test_Nexus.py has not been updated yet. I'm a little confused as to what to do though. When I run black and then "flake8 test_Nexus.py" it only shows 4 lines with trailing whitespace, but when I fix that and go to commit, the pre-commit hook complains about all the missing docstrings for public classes and functions. Glancing at some of the other users work, it looks like some of them also have classes with missing docstrings, eg. class TestSwissProt in test_SwissProt.py.

This file also has the issue of black wanting to expand all the number lists to many lines, but I'll worry about that later.

Any guidance is appreciated!

MarkusPiotrowski commented 4 years ago

Are you using the .flake8 config file of the master branch? E.g. the errors you mentioned are, afaik, turned off for the test folder.

gandalfthegrape commented 4 years ago

I'm not quite sure. When I run flake8 Tests/test_Nexus.py it does not show errors, but when I go to commit it gives me several errors that the .flake8 file should be ignoring.

svalqui commented 4 years ago

The file .flake8 has a section where list some errors that can be ignored: https://github.com/biopython/biopython/blob/6c75d2da18afdc89b6b3ef994d87c601f856ae27/.flake8#L45-L48 for example if you run flake and get this: test_Nexus.py:1168:1: D102 Missing docstring in public method you can ignore it, that relates to missing documentation, a missing docstring in a method not this issue.

if you get this: test_Nexus.py:727:46: W291 trailing whitespace then you should fix it, this is not been ignored, most likely a space in the wrong place that makes the code not PEP compliant or looks odd.

This issue relates to the styling of the code only, black makes the code look even, however in this project the results of black styling is not necessary better, for example some list are presented as one item per line making long lists hard to read, that might need adjustment, core members or contributors will advice accordingly.

I hope this helps.

gandalfthegrape commented 4 years ago

Thanks you for the advice! That makes sense, I couldn't figure out how to force a commit when it gives those errors I mentioned, but I realized I can use the --no-verify argument when I'm sure those are the only errors. I'll get working on this then.

MarkusPiotrowski commented 4 years ago

Probably, your pre-commit hook is set wrongly. Have you tried to remove and re-set it? Are you in the \biopython folder when doing the commit? I mean, the place from where you type git commit ...

peterjc commented 4 years ago

One simple possibility: It could be you had a flake8 error, did the git add, git commit - and got the warning. If you then fixed the flake8 error, and repeated git commit without doing git add first, it would still be trying to commit the bad version, not the newer fixed version. If so, do another git add, then retry the git commit.

peterjc commented 4 years ago

Many thanks to @svalqui - just merged a bunch of their pull requests for this issue which had been left pending.

martonlanga commented 4 years ago

Went through the files, to see which ones need black. 120/210 already done! Feel free to tag me if you make a PR, and I'll edit this to make it up to date.

[x] common_BioSQL.py
[x] common_BioSQL_online.py
[ ] gen_results.py
[x] pairwise2_testCases.py
[ ] requires_internet.py
[ ] requires_wise.py
[x] run_tests.py
[ ] search_tests_common.py
[ ] seq_tests_common.py
[x] test_Ace.py
[x] test_Affy.py
[x] test_AlignIO.py
[x] test_AlignIO_ClustalIO.py
[x] test_AlignIO_EmbossIO.py
[x] test_AlignIO_FastaIO.py
[x] test_AlignIO_MauveIO.py
[x] test_AlignIO_PhylipIO.py
[x] test_AlignIO_convert.py
[ ] test_AlignInfo.py
[ ] test_Application.py
[ ] test_BWA_tool.py
[ ] test_BioSQL_MySQLdb.py
[x] test_BioSQL_MySQLdb_online.py
[ ] test_BioSQL_mysql_connector.py
[x] test_BioSQL_mysql_connector_online.py
[ ] test_BioSQL_psycopg2.py
[x] test_BioSQL_psycopg2_online.py
[x] test_BioSQL_sqlite3.py
[x] test_BioSQL_sqlite3_online.py
[ ] test_Blast_Record.py
[x] test_CAPS.py
[ ] test_Chi2.py
[ ] test_ClustalOmega_tool.py
[ ] test_Clustalw_tool.py
[ ] test_Cluster.py
[ ] test_CodonTable.py
[ ] test_ColorSpiral.py
[ ] test_Compass.py
[ ] test_Consensus.py
[ ] test_Crystal.py
[ ] test_DSSP_tool.py
[x] test_Dialign_tool.py
[x] test_EMBL_unittest.py
[x] test_Emboss.py
[x] test_EmbossPhylipNew.py
[x] test_EmbossPrimer.py
[x] test_Entrez.py
[x] test_Entrez_online.py
[x] test_Entrez_parser.py
[x] test_Enzyme.py
[x] test_ExPASy.py
[ ] test_FSSP.py
[ ] test_Fasttree_tool.py
[ ] test_File.py
[x] test_GenBank.py
[x] test_GenomeDiagram.py
[x] test_GraphicsBitmaps.py
[x] test_GraphicsChromosome.py
[x] test_GraphicsDistribution.py
[x] test_GraphicsGeneral.py
[x] test_HMMCasino.py
[x] test_HMMGeneral.py
[x] test_KEGG.py
[x] test_KEGG_online.py
[x] test_KGML_graphics.py
[x] test_KGML_graphics_online.py
[x] test_KGML_nographics.py
[x] test_KeyWList.py
[x] test_LogisticRegression.py
[ ] test_MSAProbs_tool.py
[ ] test_MafIO_index.py
[ ] test_Mafft_tool.py
[ ] test_MarkovModel.py
[ ] test_Medline.py
[ ] test_Muscle_tool.py
[x] test_NACCESS_tool.py
[x] test_NCBITextParser.py
[ ] test_NCBIXML.py
[x] test_NCBI_BLAST_tools.py
[x] test_NCBI_qblast.py
[ ] test_NMR.py
[x] test_NaiveBayes.py
[ ] test_Nexus.py
[x] test_PAML_baseml.py
[x] test_PAML_codeml.py
[x] test_PAML_tools.py
[x] test_PAML_yn00.py
[ ] test_PDB.py
[x] test_PDBList.py
[ ] test_PDB_Dice.py
[x] test_PDB_FragmentMapper.py
[x] test_PDB_KDTree.py
[x] test_PDB_MMCIF2Dict.py
[x] test_PDB_MMCIFParser.py
[x] test_PDB_Polypetide.py
[x] test_PDB_ResidueDepth.py
[ ] test_PDB_StructureAlignment.py
[x] test_PDB_Superimposer.py
[ ] test_PDB_parse_pdb_header.py
[x] test_PDB_vectors.py
[ ] test_PQR.py
[ ] test_Pathway.py
[ ] test_Phd.py
[x] test_Phylo.py
[x] test_PhyloXML.py
[x] test_Phylo_CDAO.py
[x] test_Phylo_NeXML.py
[x] test_Phylo_matplotlib.py
[x] test_Phylo_networkx.py
[x] test_PopGen_GenePop.py
[x] test_PopGen_GenePop_EasyController.py
[x] test_PopGen_GenePop_nodepend.py
[x] test_Prank_tool.py
[x] test_Probcons_tool.py
[x] test_ProtParam.py
[ ] test_QCPSuperimposer.py
[x] test_RCSBFormats.py
[x] test_Restriction.py
[x] test_SCOP_Astral.py
[x] test_SCOP_Cla.py
[x] test_SCOP_Des.py
[ ] test_SCOP_Dom.py
[x] test_SCOP_Hie.py
[x] test_SCOP_Raf.py
[x] test_SCOP_Residues.py
[x] test_SCOP_Scop.py
[x] test_SCOP_online.py
[x] test_SVDSuperimposer.py
[x] test_SearchIO_blast_tab.py
[ ] test_SearchIO_blast_tab_index.py
[x] test_SearchIO_blast_text.py
[x] test_SearchIO_blast_xml.py
[ ] test_SearchIO_blast_xml_index.py
[x] test_SearchIO_blat_psl.py
[ ] test_SearchIO_blat_psl_index.py
[x] test_SearchIO_exonerate.py
[ ] test_SearchIO_exonerate_text_index.py
[ ] test_SearchIO_exonerate_vulgar_index.py
[x] test_SearchIO_fasta_m10.py
[ ] test_SearchIO_fasta_m10_index.py
[x] test_SearchIO_hhsuite2_text.py
[x] test_SearchIO_hmmer2_text.py
[ ] test_SearchIO_hmmer2_text_index.py
[x] test_SearchIO_hmmer3_domtab.py
[x] test_SearchIO_hmmer3_domtab_index.py
[x] test_SearchIO_hmmer3_tab.py
[ ] test_SearchIO_hmmer3_tab_index.py
[x] test_SearchIO_hmmer3_text.py
[x] test_SearchIO_hmmer3_text_index.py
[x] test_SearchIO_interproscan_xml.py
[x] test_SearchIO_legacy.py
[x] test_SearchIO_model.py
[x] test_SearchIO_write.py
[x] test_SeqFeature.py
[x] test_SeqIO.py
[x] test_SeqIO_AbiIO.py
[x] test_SeqIO_FastaIO.py
[x] test_SeqIO_Gck.py
[x] test_SeqIO_Insdc.py
[x] test_SeqIO_NibIO.py
[ ] test_SeqIO_PdbIO.py
[x] test_SeqIO_QualityIO.py
[ ] test_SeqIO_SeqXML.py
[x] test_SeqIO_SnapGene.py
[ ] test_SeqIO_Xdna.py
[x] test_SeqIO_features.py
[x] test_SeqIO_index.py
[x] test_SeqIO_online.py
[x] test_SeqIO_write.py
[x] test_SeqRecord.py
[x] test_SeqUtils.py
[x] test_Seq_objs.py
[x] test_SffIO.py
[x] test_SubsMat.py
[x] test_SwissProt.py
[x] test_TCoffee_tool.py
[x] test_TogoWS.py
[x] test_TreeConstruction.py
[x] test_Tutorial.py
[x] test_UniGene.py
[x] test_UniProt_GOA.py
[ ] test_Uniprot.py
[x] test_Wise.py
[x] test_XXmotif_tool.py
[ ] test_align.py
[ ] test_align_substitution_matrices.py
[ ] test_bgzf.py
[x] test_cellosaurus.py
[ ] test_codonalign.py
[x] test_geo.py
[x] test_kNN.py
[x] test_lowess.py
[ ] test_mmtf.py
[ ] test_mmtf_online.py
[ ] test_motifs.py
[ ] test_motifs_online.py
[x] test_pairwise2.py
[x] test_pairwise2_no_C.py
[x] test_pairwise_aligner.py
[x] test_phenotype.py
[x] test_phenotype_fit.py
[x] test_phyml_tool.py
[x] test_prodoc.py
[x] test_prosite1.py
[x] test_prosite2.py
[x] test_psw.py
[x] test_raxml_tool.py
[x] test_samtools_tool.py
[x] test_seq.py
[x] test_translate.py

svalqui commented 4 years ago

I'll work on test_psw.py

svalqui commented 4 years ago

I'll work on test_phyml_tool.py

svalqui commented 4 years ago

I'll work on test_pairwise2_no_C.py

svalqui commented 4 years ago

I'll work on test_codonalign.py

svalqui commented 4 years ago

I'll work on test_cellosaurus.py

svalqui commented 4 years ago

I'll work on test_Ace.py

svalqui commented 4 years ago

I'll work on test_Cluster.py

svalqui commented 4 years ago

I'll work on test_seq.py

jvfe commented 4 years ago

I'm new to contributing but I'll try working on these: test_Prank_tool.py,test_Dialign_tool.py, test_Affy.py, test_NACCESS_tool.py

svalqui commented 4 years ago

I'll work on test_CodonTable.py

peterjc commented 4 years ago

I've added this issue as a suggestion for the BCC2020 CoFest, since it is relatively straightforward to help with.

jvfe commented 4 years ago

I'll work on test_SeqIO_Xdna.py, test_SeqIO_write.py and test_PDB_MMCIFParser.py

chrisbarnettster commented 4 years ago

I'll work on:

test_DSSP_tool.py
test_FSSP.py

https://github.com/biopython/biopython/pull/3109

jvfe commented 4 years ago

Not an extensive list, but from Tests/test_* I believe these are the ones remaining (checked if someone's already working on them):

[x] Tests/test_AlignIO_convert.py
[x] Tests/test_AlignInfo.py
[x] Tests/test_BWA_tool.py
[x] Tests/test_Blast_Record.py
[x] Tests/test_Chi2.py
[x] Tests/test_ClustalOmega_tool.py
[x] Tests/test_Clustalw_tool.py
[x] Tests/test_Cluster.py
[x] Tests/test_CodonTable.py
[x] Tests/test_ColorSpiral.py
[x] Tests/test_Compass.py
[x] Tests/test_Consensus.py
[x] Tests/test_Crystal.py
[x] Tests/test_DSSP_tool.py
[x] Tests/test_Emboss.py
[x] Tests/test_FSSP.py
[x] Tests/test_Fasttree_tool.py
[x] Tests/test_File.py
[x] Tests/test_GenBank.py
[x] Tests/test_HMMCasino.py
[x] Tests/test_HMMGeneral.py
[x] Tests/test_KEGG.py
[x] Tests/test_KEGG_online.py
[x] Tests/test_NMR.py
[x] Tests/test_Nexus.py
[x] Tests/test_PAML_codeml.py
[x] Tests/test_PDB_MMCIF2Dict.py
[x] Tests/test_Pathway.py
[x] Tests/test_Phd.py
[x] Tests/test_PopGen_GenePop.py
[x] Tests/test_ProtParam.py
[x] Tests/test_SeqIO.py
[x] Tests/test_SeqIO_Xdna.py
[x] Tests/test_SeqIO_features.py
[x] Tests/test_SeqIO_index.py
[x] Tests/test_SeqIO_write.py
[x] Tests/test_Seq_objs.py
[x] Tests/test_SwissProt.py
[x] Tests/test_UniGene.py
[x] Tests/test_Uniprot.py
[x] Tests/test_align.py
[x] Tests/test_align_substitution_matrices.py
[x] Tests/test_codonalign.py
[x] Tests/test_pairwise2.py
[x] Tests/test_pairwise_aligner.py

padix-key commented 4 years ago

I will work on Tests/test_GenBank.py.

peterjc commented 4 years ago

The end is in sight. Some will be tiny changes as the file was previously black formatted but some minor edits were done which didn't quite follow the style (and we missed that because it isn't yet enforced on the Tests/ folder).

biopython / biopython

Apply black to Tests/ #2552