Harris Matrix: test first example from URAP

[lbestock]

Testing Unit FA much as #2398 tests an MKAP unit.

Initial notes:

FA-001 and FA-002 are bonded. That is a "same time as" relationship and they are not on the same level. Furthermore FA-024 is known directly (now that I updated the relations) to be later than FA-001. Shouldn't FA-001 be lower than FA-024, then?

In general the levels are very poor indicators of chronology. Only the relations between individual nodes are useful on that front, the overall needs me to move things up or down all over the place. I note only the FA-001/2 thing because I thought this should have caught that with a relationship thus defined. (This is a problem with the HM generator, too. I

There is a clear test case of the A-B-C issue in FA-025, FA-026, FA-027. If I am reading this purely for chronology, which I am, it does not bother me that this is represented as simple and linear while the physical relations were not.

[urapadmin]

Here is the new FA as an SVG:

[urapadmin]

In general the levels are very poor indicators of chronology. Only the relations between individual nodes are useful on that front, the overall needs me to move things up or down all over the place.

Is that a critique of the concept of the Harris Matrix, then I agree, or do you think some of those loci could have been placed in a better layer by the algorithm on the basis of the available information?

[lbestock]

I need to test if it would be possible on the basis of available information. But it is clear to me that this could only be used in interpretation itself, not as an illustration of interpretation, if one cannot adjust the chronological levels/vertical positions of things manually. For instance I suspect strongly enough that it wants representation that FA-056 and FA-027 are contemporary. That is not algorithmically derivable. But I would like to move the whole of the left most column (Sondage 1) down so that FA-027 is at the same level as FA-056 (in Sondage 2).

Note to myself: in terms of deriving things algorithmically, your test should be: You know that FA-005 is later than FA-002. Is there any way Kiosk should have known that? That should be complicated enough...

[urapadmin]

well, I "strongly suspect" is problematic enough. My suggestion would be that one can add interpretational relationships that would be visualized differently from the others and I can even imagine that one could add different sets of them. But lets not make it too complicated. If I give you the ability to establish an interpretational relationship in the graph that re-renders the thing (only be aware that re-rendering can change the whole graph substantially) then you could add the algorithmic data that makes the nodes move to the layers you suspect they belong.

[lbestock]

Well but that is just it, right? Is the HM a representation of the known certain chronological implications of stratigraphic relations? Or is it a representation of the interpretation of chronological events? I want to be able to play around with an interpretational relationship and see what it does, but is that really complicated on your end?

I am currently struggling with an interesting case. FA-006 is a wall. All the walls were built before the floor was put down. The floor was put down in two phases, FA-023 is a mud layer, and then FA-024 is the bricks set in it. In the case of FA-006 we did not have a cut perpendicular to the wall, so we saw this only from the top, we could only record that FA-024 abuts the wall FA-006. I know from the relationship of those two floor phases to one another and how they must have been constructed, and can see in relation to other walls, that there is no way FA-023 can be the same date as FA-006 - the wall had to be there before any of the floor was built. I can only observe it for the top layer in this case. So I have an HM with FA-006 contemporary to FA-023, and that's not wrong algorithmically - they are both earlier than FA-024. But it is wrong and I know it to be so.

The relationship of the FA-024 floor to the walls is the place where I can best test the A-B-C issue. On a purely chronological level it's right everywhere else. But those are the major architectural phases of this unit and they touch each other and I really have to remind myself no, this does a different thing than a section drawing does.

[lbestock]

So now I have two interpretational relations I want to input if given that opportunity: FA-027 is the same time as FA-056. FA-006 is earlier than FA-023.

(I am editing this comment to reflect all the interpretational relations I want to put in, so I don't have to sort through the whole ticket later:

FA-027 is the same time as FA-056. FA-006 is earlier than FA-023. FA-006 is earlier than FA-004. FA-039 is later than FA-002. FA-039 is earlier than FA-005. FA-050 is later than FA-002. FA-050 is earlier than FA-005. (Those last four should have a major difference on the HM because they reflect the fact that I interpret the walls of the room as having been built at different times and with leveling fill layers between - the walls were terracing. I want to see this with and without those FA-050 interpretational ones in there! Not clear to me if I need the last two algorithmically - might be enough to just get the FA-039 ones in there.)

To play with (I thought it was possible but was not sure, and it would rearrange a few things on the HM): FA-042 is equivalent to FA-042.

Oh also to play with: FA-027 and FA-046 were thought at one point to potentially be the same layer. I need to try that - to my eye looking at the current HM that is stratigraphically impossible but I am not confident in that at all and want to know what happens if it is the case.

[lbestock]

I would also add interpretationally that: FA-006 is earlier than FA-004.

That is a less secure interpretation, however, so wow danger.

I still really like this wonderful puzzle of a unit. It was so much fun.

[urapadmin]

Well but that is just it, right? Is the HM a representation of the known certain chronological implications of stratigraphic relations? Or is it a representation of the interpretation of chronological events?

Or are there archaeologists with "formal training" in HM who do have expectations? Well, I don't know. But apart from that we can decide ourselves what we want it to be and how we want it to work. To me it seems much more useful as a tool to study your stratigraphy than as a somewhat static picture that tells you the stratigraphy (because it can't in part because there are many loci that cannot be put on one or the other layer). So playing around with it (coloring it, marking it, enriching it with other data from the record) would be interesting.

There is just one major caveat from my point of view: if I give you the opportunity to play around with your stratigraphy you might be tempted to make it fit and add interpretational relationships until it is more plausible than it actually should be.

But one can battle that part at least by visually distinguishing between relationships that have been recorded in the field or found otherwise physically compelling and suspected (or wished for :o)) relations added later.

[urapadmin]

perhaps somebody wants to read the manual? One can get it for free at http://harrismatrix.com/about-the-book/ in many languages 🌵

@luizaogs has added thoughts of the same nature in #2398. Let's discuss the "interpretational relations" here in FA for the time being.

[luizaogs]

I like the idea of interpretational relations :)

[urapadmin]

While you guys are analyzing this please try to distill a few good archaeological cases out of this that show what I think is the main weakness of the harris matrix: Seemingly false precision. While I have to place a node (locus) on one of the layers often enough the data does not require it to be on exactly one and not any other layer. So the node seems to be contemporaneous with some other node but the data is not that clear. If was for a while thinking that one should draw the corridor of layers in which a locus exists relative to other layers. Could be a shaded background or so. But I think the Harris method does not do such a thing.

So if you run into actual archaeology that reveals the problem that I only see theoretically please write it down so that it can be explained in the class room.

[luizaogs]

Yeah I noted a few instances where I thought this had happened in the AA matrix in #2398 but I'll note more if I see them.

[urapadmin]

💤 Time for me. But this is fun.

[lbestock]

Musing by the fire. Yes, the false precision is an issue and a way to capture the imprecision would be good. However. I was initially annoyed that things that are unknown are landing at the latest of the places they could be, since I often knew or suspected they were earlier, and I am not annoyed by that anymore. On the contrary, if we can't find a way to indicate imprecision, then things definitely belong on the latest possible position the algorithm returns. Why? Because the implications of dating something too late are usually much smaller than the implications of dating something too early. When someone makes a mistake too early they write books rewriting history and huge energy is needed to correct the mistake. Dating something too late usually just means you ignore it for longer than warranted. I am reminded of how I teach students to clean loci and how uncomfortable they are scraping the top of something away before beginning to collect from it, and how the division between those who get and value the problem of contamination and those who don't divides those who have a prayer of doing this from those who don't. Also my FA-006 example reminds me that "archaeology is destruction" is not a bad thing. If we had taken out FA-023 and FA-024 up to the wall that would no longer be ambiguous in the records because we could have documented the relationship instead of inferred it.

I want interpretational relations for the HM. But where do they exist and how and how aside from the HM would they be seen and recorded? I'm very wary of having them in the recording in a way equivalent to the observed relations; the slippage where something recorded becomes something extant is scary to me. We have always been careful to separate the observation and interpretation for a reason. @urapadmin do you mean to give us these in a way that changes the matrix only, or that changes the recording?

[lbestock]

Another missing relation in the data that will make a difference: FA-009 is above and thus later than FA-024.

(Take a look at FA-010 and FA-011 - if they are not contemporary with FA-009 something is weird, though there should not be any algorithmic way of determining that other than that they are all directly on top of FA-024.)

[lbestock]

Yeah wow. FA-017 is later than FA-008 and FA-009 (abutting); and FA-024 (above). Shoddy recording.

[lbestock]

(I want a quality control rule that nothing except bedrock or topsoil can possibly have only one locus relation, and even that would be suspect.)

I fixed those last ones but you might want to wait until I have gone through everything to run the HM again.

[lbestock]

Last not totally coherent thought before I go to bed: I have been thinking of ambiguity in the places where there is a long vertical line. Those are obvious - the node can slide along that line. More insidious is that there is ambiguity in places that are denser, as in above, where FA-006 and FA-023 are on the same horizontal line and earlier than FA-024. Those are all short lines, they don't jump out at you as a place of ambiguity, but we know there should be three chronological steps there not two, so there is a whole horizontal section missing (not algorithmically, but archaeologically). So it's not just where things are phasing-wise, it's even how many phases there are, that get confused in that way. In a more minor way the case of equivalence between FA-027 and FA-05 is similar. There there would not be an increase in overall number of phases, but there needs to be a longer vertical line somewhere (where?) in the relations between layers in that sondage to get those into the same phase.

[lbestock]

Already with FA and certainly with anything that has more loci: an absolutely ideal HM UI would include the ability to search for and highlight a locus. I am doing so much going back and forth (I have the recording system open on two devices and sheets of paper flying all over the place too), so it's not just going from the matrix to the recording. I need to go the other direction, too. And then it takes me ages to find my locus.

[luizaogs]

Already with FA and certainly with anything that has more loci: an absolutely ideal HM UI would include the ability to search for and highlight a locus. I am doing so much going back and forth (I have the recording system open on two devices and sheets of paper flying all over the place too), so it's not just going from the matrix to the recording. I need to go the other direction, too. And then it takes me ages to find my locus.

Oh yes, I second this if that’s possible to do. Even with AA yesterday which is much less complicated I kept losing loci and getting a headache going back and forth.

I mean, actually, an absolutely ideal HM UI would not only allow me to search for loci but also to go to those loci by clicking on them. It sounds like this will be in Kiosk rather than the recording app, yes? Is there a way to connect it to Q&V to pull up locus information that way?

[lbestock]

A case of what I might call compound false precision that is interesting: Several small but disconnected walls on the west side of FA are almost certainly all of the same chronological phase - they defined the bins in which grain was stored. I think I can see in looking at the matrix that there is no actual reason for them not to be on the same level, and indeed three of the five of them are. But the two that are not on that level are very not on that level, and not on the same level as one another. The matrix has not misinterpreted any of the entered archaeological relations incorrectly but it has made choices about where to put things that pretty seriously distort the sense because they seem precise and different. The loci in question are: FA-008 FA-009 FA-010 FA-011 FA-034

I can add these to the "interpretational" list above, but I note them separately here because I think this is what you are asking for in terms of false precision?

A much more straightforward case is FA-058. Again no way for the algorithm to know. That is on the same level as FA-019, just for instance, and FA-058 is ancient and FA-019 is a modern looter's pit, so definitely not the same phase. But there is really no way an algorithm could know this. FA--58 was ancient but it is not a well-networked locus, being a floor on top of a floor and already exposed to the sky when we found it.

[urapadmin]

Okay, while we get many ideas on how to implement a Harris Matrix as a more interactive tool here, what I still need before I can do anything is: Are the chronological relationships in FA (and AA) correct as far as the evidence allows an algorithm to infer chronology?

[urapadmin]

A case of what I might call compound false precision that is interesting: Several small but disconnected walls on the west side of FA are almost certainly all of the same chronological phase ...

FA-008 FA-009 FA-010 FA-011 FA-034

I can add these to the "interpretational" list above, but I note them separately here because I think this is what you are asking for in terms of false precision?

A much more straightforward case is FA-058. Again no way for the algorithm to know. That is on the same level as FA-019, just for instance, and FA-058 is ancient and FA-019 is a modern looter's pit, so definitely not the same phase.

• [ ] I will look into these examples and if it turns out to be random, I will see if there is any option for me to tweak the algorithm so that it consistently puts them up (which it should be doing). But I have very limited influence on the positioning of the nodes.

When you are finally done with FA and are absolutely convinced that the locus relations in the recording database are correct as far as possible (no more shoddy recording) I will go through the not insubstantial labour again and put them in the HM composer to get a final opinion on what that one spits out.

[lbestock]

On a sense level (I still have not caught this out as having any errors in applying the relations) FA-017 needs to be later than all those little walls just listed above.

[lbestock]

I like it, though. And am definitely a convert. Doing it this way I caught so much that I had not caught when a couple of years ago I really tried to work on the stratigraphy of this unit.

[urapadmin]

I just feat that you work on it currently in a way you won't again. Next time you'll just trust what you see :o) But anyway, while it isn't perfect and one mustn't take it as that, handled properly I think it can be a huge help.

[lbestock]

Back over to you - my changes are synchronized.

[urapadmin]

Not our unit, but:

WARNING: 2 locus relations still have issues that cannot be fixed automatically: INFO: BA-020 other BA-001 INFO: BA-024 other BA-023

[lbestock]

I'm not so sure of that. Because it isn't anything in and of itself really, it is a guide to understanding the loci. I would never use it except that way I think - it tells me what order to look at my loci when I am trying to write a chronological history of what happened in the unit (from first to last in the past, not what happened archaeologically, there I use the narratives). So I might not test it the same way, but I also wouldn't write conclusions just based on it and so I think I will catch stuff again. I really like it. Whereas I always thought making one would be worse than eating broken glass, so for me at least having it generated and then checking is ideal.

[lbestock]

Yeah let me take care of that so we can get a clean synchronization.

[urapadmin]

Yeah let me take care of that so we can get a clean synchronization.

I have already downloaded a backup. But it does not affect FA

[urapadmin]

Here is your new FA. this time it is all squiggly. That is the only way to make my off-the-shelf layouter draw edges from the bottom side to the top side. Personally, I don't like the squiggliness and I think it makes it harder to read. So I still would invest the time to invent my own orthogonal edge drawing algo. But I wanted to share this at least and get your opinion. Apart from that, this is your new data in graph shape:

[lbestock]

Hm. I'm interested to note that still only two of the three sondages, which are pretty straightforward, show up here in an immediately obvious way. The first time I thought it didn't because Allie's recording was inadequate but I rather expected the other one to jump out now, too. Though thinking about it, the other two sondages hit things earlier than the walls. Cool. Not your concern.

The algorithmically correct false precision about the small cross walls is definitely still a thing, so I want my interpretational relations.

I don't like the swooshy lines either. But I prefer the rectangular nodes and the arrows that hit the nodes this way, bottom to top only.

[urapadmin]

As for the visualization: I suspect that is @luizaogs opinion, too, right? In that case I'll scratch my chin a lot but in the end, if I draw the whole thing myself (currently I just show an svg somebody else rendered for me) I can do more in terms of coloring, adding new lines, etc. We'll see if I can achieve that.

[urapadmin]

@luizaogs, @lbestock : For both of you a tip: If you open the svg in a browser you can user ctrl+f (who knows what it is on a Mac?) and search for an identifier. That's how I do it.

[lbestock]

Oh hey look. FA-005 is now later than FA-002.

[lbestock]

It won't even let me open that in a browser.

[lbestock]

Oh. It will let me open it in a browser if I access it through my email, where your comment comes with a "view on web" link that I don't get when I am in GitHub itself. Confusing. And then I can use command+f to find an identifier indeed.

[urapadmin]

Oh. It will let me open it in a browser if I access it through my email, where your comment comes with a "view on web" link that I don't get when I am in GitHub itself. Confusing. And then I can use command+f to find an identifier indeed.

Isn't "in GitHub" in the browser? In that case I just click on the image and get it bigger in a new tab and can search, too. Or I download the image (right click, save image as) and work with it offline or .... There are so many options, particularly when you use the context menu.

[luizaogs]

As for the visualization: I suspect that is @luizaogs opinion, too, right? In that case I'll scratch my chin a lot but in the end, if I draw the whole thing myself (currently I just show an svg somebody else rendered for me) I can do more in terms of coloring, adding new lines, etc. We'll see if I can achieve that.

Visualization in terms of rectangular boxes with arrows that appear from bottom to top only? Yes, please. The other one gives me a headache.

[urapadmin]

As for the visualization: I suspect that is @luizaogs opinion, too, right? In that case I'll scratch my chin a lot but in the end, if I draw the whole thing myself (currently I just show an svg somebody else rendered for me) I can do more in terms of coloring, adding new lines, etc. We'll see if I can achieve that.

Visualization in terms of rectangular boxes with arrows that appear from bottom to top only? Yes, please. The other one gives me a headache.

and this was about the squiggly lines (new in the latest FA above)

[luizaogs]

Oh yeah, don't love them. But better to have them than the ovals and arrows coming into nodes from all directions, I think.

[urapadmin]

A case of what I might call compound false precision that is interesting: Several small but disconnected walls on the west side of FA are almost certainly all of the same chronological phase - they defined the bins in which grain was stored. I think I can see in looking at the matrix that there is no actual reason for them not to be on the same level, and indeed three of the five of them are. But the two that are not on that level are very not on that level, and not on the same level as one another. The matrix has not misinterpreted any of the entered archaeological relations incorrectly but it has made choices about where to put things that pretty seriously distort the sense because they seem precise and different. The loci in question are: FA-008 FA-009 FA-010 FA-011 FA-034

I can add these to the "interpretational" list above, but I note them separately here because I think this is what you are asking for in terms of false precision?

A much more straightforward case is FA-058. Again no way for the algorithm to know. That is on the same level as FA-019, just for instance, and FA-058 is ancient and FA-019 is a modern looter's pit, so definitely not the same phase. But there is really no way an algorithm could know this. FA--58 was ancient but it is not a well-networked locus, being a floor on top of a floor and already exposed to the sky when we found it.

Okay, I have looked into this. This is indeed the algorithm trying to avoid line crossings. And since there is no compelling reason to keep FA-010 on the same level as FA-009 it moves them around. There is nothing I can do about that. The layers don't mean anything It is really only the edges that show you if something is earlier or later. If there is no edge between to nodes you cannot know anything about it. I would have preferred an algorithm that is consistent in putting a node always on the layer closest to the nodes predecessor or successor but such an algorithm is not readily available. Interpretational relations are the only solution I can think of. E.g. one can cluster nodes if one is sure that they are contemporaneous:

If I cluster them they also stay together on the horizontal axis.

[lbestock]

Oh look, I missed yet another relation: FA-008 also abuts FA-002. The clustering goes some way towards making it look more like the actual stratigraphy.

It occurs to me that the reason I am uncomfortable with the false precision in dense places like this is because there is no way to visually distinguish when the precision is false or not. FA-001 and FA-002 are so contemporary that they're bonded, but they don't look any more certain than any other two loci that might be just accidentally on the same line. The long vertical lines for some loci are such helpful indication that the precision is false. I wonder if I would like a longer more vertical matrix that always added another phase/horizontal line if it didn't know for sure that things were contemporary, so that you could trust the horizontals. And then the process of interpretation would be more straightforwardly determining which of the extra phases are not real by seeing if any things that are in separate lines belong in the same lines. It might be less confusing to me than this.

[urapadmin]

alright. I close these and work on solutions. We open new tickets when I have something to show

[urapadmin]

Oh look, I missed yet another relation: FA-008 also abuts FA-002. The clustering goes some way towards making it look more like the actual stratigraphy.

It occurs to me that the reason I am uncomfortable with the false precision in dense places like this is because there is no way to visually distinguish when the precision is false or not. FA-001 and FA-002 are so contemporary that they're bonded, but they don't look any more certain than any other two loci that might be just accidentally on the same line. The long vertical lines for some loci are such helpful indication that the precision is false. I wonder if I would like a longer more vertical matrix that always added another phase/horizontal line if it didn't know for sure that things were contemporary, so that you could trust the horizontals. And then the process of interpretation would be more straightforwardly determining which of the extra phases are not real by seeing if any things that are in separate lines belong in the same lines. It might be less confusing to me than this.

A Harris Matrix, I think, never tells you anything about the loci on the same layer. It only tells your something about loci connected with edges. The layers themselves (not stratigraphic layers, I mean the horizontal strips nodes seem to share) get misinterpreted. I don't see how to get out of that, though. When you write that you don't know if a node being on a layer is false precision then the answer is almost always: Yes it is false precision if your read it as chronological There is no reason for any of the nodes to be on a certain layer. How many layers even exist depends on the density of the stratigraphy in a certain region. I can't imagine how even on paper or with the HM composer this could be any other way. I suspect that the intuition one has trouble to suppress is simply a false intuition. But I also think that the absence of drawing contemporaneous relations is in the end part of the trouble. I will think about ways to overcome that (perhaps with colour or so).

[urapadmin]

If you think FA now has the proper relations I would feed it once more to the Composer and then you can muse about why that spits out a more useful or less counter-intuitive graph, if at all.

[urapadmin]

small update today today: FA with the actual contemporary relations marked by light grey arrows:

(@lbestock: please answer the question above when you are ready!)

[luizaogs]

Thinking of the trouble I had with the AA matrix, that seems very helpful.

[lbestock]

I do have a question about why in the cluster one FA-008 is treated as having a different relationship to FA-002. Or at least, it is the only one of the cluster whose abutting to FA-002 is not rendered, and I thought I must have missed the relation but at least in the current version (so many versions, one of the problems) it is already in there.

So. I like the grey arrows. Not too intrusive, provide the answer as to which things are actually known to be contemporary. I dislike that it crosses FA-023 up there - @urapadmin you just added these, or were they taken into account when rendering? I am ready to compare this with the proprietary HM generator, so you can go ahead and feed that beast.