Key error using verbose 0 in console table

[PonteIneptique]

Corpus ran on : https://github.com/lascivaroma/pompei-inscriptions/

Console output :

hooktest . --workers 3 -s epidoc --verbose 0 --console table --f cil004-00100
>>> Starting tests !
>>> Files to test : 203
...................................................................................................X................................XX......XX.........................X...................................
Traceback (most recent call last):
  File "/home/thibault/dev/pompei-inscriptions/venv/lib/python3.4/site-packages/HookTest/test.py", line 826, in cmd
    status = test.run()
  File "/home/thibault/dev/pompei-inscriptions/venv/lib/python3.4/site-packages/HookTest/test.py", line 435, in run
    self.end()
  File "/home/thibault/dev/pompei-inscriptions/venv/lib/python3.4/site-packages/HookTest/test.py", line 535, in end
    if unit.units["Passage level parsing"] is False:
KeyError: 'Passage level parsing'

[PonteIneptique]

@sonofmun Would you mind having a look at that when you take on the different output ?

[sonofmun]

I don't get this error when testing the First1K repo on --verbose 0. I will take a look at the repository later in the week to try to track down the problem.

[PonteIneptique]

The bug happens only with -f set

[sonofmun]

It also works on First1K with -f set:

hooktest -w 8 -s epidoc --countwords --console table --verbose 0 -f tlg0086 ~/OpenMigne/First1KGreek/First1KGreek/
>>> Starting tests !
>>> Files to test : 56
........................................................
+------------+-------+-------+--------------+
| Identifier | Words | Nodes | Failed Tests |
+------------+-------+-------+--------------+

>>> End of the test !

+----------------------+---------+
|   HookTestResults    |         |
+----------------------+---------+
|     Total Texts      |    32   |
+----------------------+---------+
|    Passing Texts     |    32   |
+----------------------+---------+
|    Metadata Files    |    24   |
+----------------------+---------+
|   Passing Metadata   |    24   |
+----------------------+---------+
|       Coverage       |  100.0  |
+----------------------+---------+
| Total Citation Units |  4,358  |
+----------------------+---------+
|     Total Words      | 580,882 |
+----------------------+---------+
|     Words in MUL     |  36,608 |
+----------------------+---------+
|     Words in LAT     |  3,302  |
+----------------------+---------+
|     Words in GRC     | 540,972 |
+----------------------+---------+

[PonteIneptique]

Can you remove countwords ?

[sonofmun]

Still works:

 hooktest -w 3 -s epidoc --console table --verbose 0 --f tlg0086 ~/OpenMigne/First1KGreek/First1KGreek/
>>> Starting tests !
>>> Files to test : 56
........................................................
+------------+-------+--------------+                                                                                        
| Identifier | Nodes | Failed Tests |                                                                                        
+------------+-------+--------------+

>>> End of the test !

+----------------------+-------+
|   HookTestResults    |       |
+----------------------+-------+
|     Total Texts      |   32  |
+----------------------+-------+
|    Passing Texts     |   32  |
+----------------------+-------+
|    Metadata Files    |   24  |
+----------------------+-------+
|   Passing Metadata   |   24  |
+----------------------+-------+
|       Coverage       | 100.0 |
+----------------------+-------+
| Total Citation Units | 4,358 |
+----------------------+-------+

[PonteIneptique]

Wait, it does not work with no finder as well...

[sonofmun]

If you want, I will clone the repo and figure out what is causing the problem. Not today, however (staff meeting soon).

[PonteIneptique]

Oh sure. I am just trying to figure it out also a little. I wonder if it could be because I have only one level ?

[sonofmun]

I have lots of files with only one citation level in First1K. What if you try it with another workgroup or work identifier? Maybe there is a problem with one of the files in that workgroup.

[PonteIneptique]

The issue is tied to my dot issue :

/data/cil004-00000/00001/__cts__xml {'File parsing': True, 'Metadata availability': True, 'Naming Convention': True, 'MyCapytain parsing': True, 'URNs testing': True}
Traceback (most recent call last):
  File "/home/thibault/dev/canonicals/pompei-inscriptions/venv/lib/python3.5/site-packages/HookTest/test.py", line 827, in cmd
    status = test.run()
  File "/home/thibault/dev/canonicals/pompei-inscriptions/venv/lib/python3.5/site-packages/HookTest/test.py", line 435, in run
    self.end()
  File "/home/thibault/dev/canonicals/pompei-inscriptions/venv/lib/python3.5/site-packages/HookTest/test.py", line 536, in end
    if unit.units["Passage level parsing"] is False:
KeyError: 'Passage level parsing'

Have you tried with a fresh hooktest install ?

[PonteIneptique]

I have it on two different machines...

[sonofmun]

Not yet. Testing it now.

[sonofmun]

Also works with a fresh hooktest install (i.e., new virtualenv where I only ran pip install hooktest.)

[sonofmun]

Does First1K test successfully on your installations?

[PonteIneptique]

This is so weird...

[sonofmun]

I want to point out that I haven't tested your repository yet. I have only run the tests with the same parameters on First1K.

[PonteIneptique]

Ok. I found the bug. It should do the same for you

If you run this from inside the directory you test, using "." as the current dir, the code in https://github.com/Capitains/HookTest/blob/3537e83184d7271a6822453c039556c348f60290/HookTest/test.py#L936-L940

    def directory_replacer(self, data):
        if self.repository:
            return data.replace(self.directory, self.repository)
        else:
            return data.replace(self.directory, "")

Then any dot in the name is replaced by ""

[PonteIneptique]

The fix would be

    def directory_replacer(self, data):
        if self.repository:
            return data.replace(self.directory, self.repository)
        elif self.directory != ".":
            return data.replace(self.directory, "")
        else:
            return data

[sonofmun]

That recreates the problem for me. And it throws the error as well with --verbose 5 and --verbose 10. I will make that change right now.

[sonofmun]

OK. I have it working now. I will write a test for it tomorrow and make the other changes necessary for bumping the version.

[sonofmun]

I think there might still be a problem here, though I must say I don't understand this function 100%. If directory is "." and UnitLog receives a value for repository, then all of the . will be replaced with the repository name. E.g.,

log = HookTest.test.UnitLog(
            directory=".",
            repository="First1K",
            name='tlg0000.tlg000.tlg00.xml',
            units={},
            coverage=100.0,
            status=False
        )

produces the log.name of tlg0000First1Ktlg000First1Ktlg00First1Kxml. So should the test to see whether ''self.directory == "."come beforeif self.repository`?

[sonofmun]

Ah, but now that I look, I think this case will be taken care of by https://github.com/Capitains/HookTest/blob/master/HookTest/test.py#L252-L263. So I don't think this will be a problem.