Is the contour extraction working?

[Armand1]

I have looked at the contours. I wanted to overlay the the contours by shape classes but actually failed to do that since I could not flip them all the right way. But I have scaled them roughly within each shape and smoothed them a bit.

My feeling is that they're not bad at all. But one thing is very clear: we need to know the line of reflection of the x-axis of each vessel. That's because quite a lot of information is in their height-width ratio (y/x). So the edge at the top and the bottom should run out to the same extent, and be truncated at the line of reflection (if that's what it's called, but I think you know what I mean).

I am pretty sure that, without that, there is no hope of getting the registration right.

[LittleAri]

They definitely need improving!

I definitely agree that the the edge at the top and bottom should be running out to the same extent and this is the next thing I will edit within the contour extraction code!

I've been doing the "flipping" in MATLAB, just before registration takes place. So in case it's useful, the MATLAB data file would have the reflected contours. However, it would make more sense to do the "flipping" earlier so I'll add that to the contour extraction code next. Thanks for the reminder!

[Armand1]

Flipping earlier would be a great help. MATLAB's not much good to me. Might as well be plain about my limitations: R good, Python barely, MATLAB not at all. So I won't be running your code, but only looking at csv output.

[LittleAri]

Hello! Apologies for the late response!

I've updated the extraction code. Right now, the code tries to find the centre of the pot. It then either extends or cuts the top and bottom of the contour so that it starts and ends at the x coordinate of the centre of the pot. Then, if needed, it reflects the pots, so all the pots should now have their right contour outlined.

It's not perfect! For some of them, the centre isn't really in the centre, and for 1-2, the centre is no-where near the actual centre (mainly due to poor quality of image). You can see the contours in the folder "updated_original_pots_with_contour". I've attached a few here as an example.

The red contour is the old previous contour, the green contour is the updated one, and the yellow square is the centre the code found.

I've also rerun all the analysis with these new contours and have created new CSVs, datafiles and plots, all of which should have the same labeling as before, except for the word "updated_" beforehand, so we can distinguish them from the previous ones.

I'm going to try and improve the code that finds the centre, but thought I'll share these for now since they do seem to look better. 👍

[Armand1]

I had a quick look. They seem pretty awesome to me! We can begin to see real variation in shape within classes. Take, for example, the first class: alabastrons --- one of them is really taller than the others (I think it's the one illustrated above as a matter of fact). So that's important! But the amphora-a, amphora-neck, and kraters are lining up remarkably well! Looking at the individual contours around the photos I see some errors. But, bear in mind, that at some point we will want to say "good enough" in the hope that errors will just average out over large sample sizes. We're not classical pot people who obsess about individual pots!

[LittleAri]

Oo these are looking nice!! Even the chalices!

I did have some other ideas that I've not yet tried, in order to try and get more accurate centres. So I'll try them at some point but don't worry, I won't obsess over it too much! 😄

[Armand1]

Karcher means of a few select shapes

[Armand1]

I have made a new contour extraction test dataset. It contains 821 image files. These have been selected from athenian (geometric, black-figure, red-figure), corinthian, boeotian and lucanian, east-greek/asia-minor vases. For each fabric, and shape-class, I have selected 5 (the number varies a bit) vases that should be of the same type. For example, there are 5 amphora_a_athenian_black-figure. I have also put a selection of Mantamados pots in there. (I am very curious to know how modern Greek pots compare to ancient ones --- and they're gold standard images.)

These vases are "clean". There is only one image per id. Each image has been vetted by me, so that I think you should be able to extract a contour from it. If I thought there were difficulties, they have been rectified in photoshop. For example, lids and stands have been removed. If I thought that the vase was poorly resolved against the background, it's been cutout and put on a better background. And so on. Of course, I do not know exactly how the contour extraction behaves: but that is what we need to find out.

I have renamed the vases so: shape_generic_shape_specific_fabric_style_1_fabric_style2_id.jpg" This is so that you know instantly what they are.

This is not the total diversity of vases. Athens certainly contains other shapes that we don't have here (since we've only sampled it, we don't have a completely curated dataset); and this is missing various fabrics. So we might still encounter all sorts of weird handles and what not in the future: but it's quite a large chunk of the diversity out there. And, my idea is that we solve these --- and then add to it as more and more vases are cleaned and curated, until finally we are in a position to run contour extraction on everything.

Here is the dropbox containing the vases

https://www.dropbox.com/sh/liwdtvtdquubtad/AAAebvytl_EXMhFGuLbd7MEaa?dl=0

[Armand1]

Arianna has provided contours of the 821 images. She plotted the contours over the images so we can evaluate them. That's great. It's very clear to me that the contours of every vase will need to be evaluated by eye for a good fit before being admitted into a final dataset. This is, of course, a lot of work: but actually it's not that hard. You can check 3000 per hour or so. So it's a few days work for the entire dataset, and we can do it gradually.

But the following is apparent.

1) I really need the contours plotted over the original image (or perhaps a size-standardized) version rather than the binarized version. That's because in the binarized version it's sometimes hard to see what is --- or was --- really there.

2) The contours are not smoothed. But Arianna says that, some vases --- with additional smoothing (her Rank 2) --- will be OK. Maybe so, but I can't evaluate that on the unsmoothed contours. So, I need to see the final, smoothed contours over the original images.

3) There are no contours over the really bad ones (Rank 3). So I can't judge where they went wrong (was it shadows, crap background, awkward handles or what?) I need to see those contours over those images too.

Once we identify images with bad contours, we can decide what to do with them. The options are basically: (i) discard; (ii) fix in photoshop and return for another round of contour finding.

[Armand1]

Of the 821 images, Arianna graded 591 (72%) "Rank 1" --- good contours. I've been through them too. A bit more conservative than her, I would say 539 (66%) are good. Those are the ones I am going to analyze now. But once I see the smoothed contours I am sure that by including her Rank 2 we'll get up to about 75% good.

That means we will have to chuck or fix about 25% of our images (after initial quality filtering) perhaps 5,000 out of, say, 20,000 images. Not bad at all. Bear in mind that we can average groups of images as well (Karcher means I guess). So that will help reduce noise. What groups we average is another question.

[Armand1]

These are size-standardized plots of the 539 vases that I graded as good (a subset of Arianna's Rank 1). They look pretty nice. But are the csv files smoothed? Some of them look quite jagged...

[LittleAri]

Hey!

1) I can't think of an easy way to create plots with the original images and the contours superimposed in the exact right place. They're not the same sizes necessarily. This is probably due to the fact that when I binarize the photos, I save them as a plot and then find the contour from that new image, which is usaully smaller than the original image. Instead, I could create two subplots, one with the original image on the left side, and the contour on a white background on the right, for example. Would that be helpful? Though let me know if you have any other ideas on how to do this! 2) The CSVs don't contain the smoothed pots, but the MATLAB datafile does. So tomorrow when I'm back in the office, I'll upload the smoothed version of all the contours in CSV format. Apologies for not having done this earlier! 3) The issue with uploading bad contours was that there were a lot. Essentially, the reason for this is because I didn't want to give up on the pots. So if there was a bad contour, I then tried to run it again but without image-binarization for example. Or in some cases (Group 2 pots in particular i.e. pots with Bell-krater-like handles), I may have tried estimating a new location for its handle if it didn't work the first time. So some Rank 3 pot images might have multiple bad contours. If it's helpful, I could upload all of the bad contours for each Rank 3 image, and then you can potentially see where it is going wrong. E.g. with the two cups below: by just looking at the first image, you might think that this pot was classed as Rank 3 due the patterns all over the pot. However, when looking at the lower image, it becomes clear that the main issues are actually with the bottom of the pot - otherwise the algorithm should have been able to find the contour of the binarized image. Is this something you would like me to upload?

I did try and be strict but I'm not surprised that I was more lenient. 😄 But 66% is quite good! Do you have a list of the ones that you've demoted from Rank 1? It would be good to have a list of these so I can test new smoothing techniques on them.

Thanks for checking all of these out, Armand!

P.S. Checking 3,000 or so pots in an hour is seriously impressive!

[Armand1]

Great, thanks. Here is what I think:

1) CSVs that you give me should be of smoothed versions 2) Any distance matrices that you give me should be of smoothed. 3) How about original on left, with binarized+contour on right?

How does your contour-evaluation work? For each image, do you find contours with all four algorithms, then check them all manually, identify the best one, and then evaluate rank? Or do you generally apply one contour-finding algorithm, and only when that fails try all four?

You see, I am trying to figure out an efficient pipeline. Ideally, you'd give us a single best fit for each image; we (Han and I) would judge whether it's good; if it's not good we fix and return to you for another round of contour fitting. I want to avoid having you judging contours (not because you can't, but because it's more efficient if I do it and then fix immediately).

Would the following work? For each image, you find a contour with all four algorithms. Let's call them: algorithm 1 --- "Blue"; algorithm 2 --- "Red"; algorithm 3 -- "yellow"; algorithm 4 -- "Green". Draw all four smoothed contours over each binarized image (original on the left). I look at each composite image and sort into a blue folder, red folder, yellow folder, green folder according to the best fit, and and a "bad" folder (all are bad). "bad" images are fixed and returned to you for refitting.

3000 ph is an exaggeration. 1500 more like it!

[LittleAri]

1. Sounds good! Will send this over to you soon!
2. The distance matrices should already be of the smoothed versions of the contours. But if I update the smoothing techniques, I'll update these matrices too.
3. Okie dokie, that could work nicely, potentially! 🙂

That sounds like a good plan! The only thing is that I don't use each algorithm for each pot. Which/how many algorithms I use for each pot depends on the group they're in (based on the groups I denoted in the PDF document):

• For pots in Groups 1,3,5, they only use one algorithm each. And I can switch off image binarization in those, so those ones should have 2 contours each.

• For Group 4 pots, they have one main algorithm like the other groups. However, there is a subset of those pots (cups with handles right on the top of the pot) that use a slightly different algorithm, specially designed for those kinds of pots). But again, Group 4 pots would also have 2 contours each.

• For Group 2 pots, they have one main algorithm. So again, I would run it twice, once with image binarization and once without. But in addition to this, for those pots who still don't have nice contours, I try extracting contours using another Group-2-specific-algorihtm. This algorithm is one of the old ones that I had created that can check for handles in three different locations of the pot ("top", "middle", "bottom"), And again, you can choose to binarize the image. So most Group 2 pots would have only 2 contours. But some of them will have 4 (2 from the main algorithm and 2 from the older algorithm).

[LittleAri]

Just to add, only Group 2 and Group 4 pots might share an algorithm. The Group 2 subset with their handles right on the top e.g. Skyphos Cups, would use the same algorithm as Group 2 pots.

[Armand1]

This is very complicated. Do you have an automatic way of determining which algorithm/s to use on any given pot? If not, how are you going to scale this up to 10^4 pots?

[LittleAri]

Currently whenever I receieve the pot images, the first thing I do is sort them into the groups. From there, it's not too complicated since I know which algorithms would be used for each group. The hard part is the sorting, but that doesn't usually take too long. You could definitely do a couple thousand in an hour, since it's easy to see whether something has, for example, no handles, an amphora-like handle, a handle on top etc.

In future I would like this part to be automated. I have played around with this many times. E.g. checking the name of the file and then that sorts the pot and decided on the algorithm automatically. However the trouble with going by name is that some pots in the same class have different handle styles e.g. some chalices have no handles at all yet some of them do. Stephen and I also played around with the idea of using all algorithms for all pots and then creating an algorithm that will detect the bad contours amongst all of these. For this case, I tested whether registering the pots and taking a distance will help decide the best pot but it's not that simple. So I put that plan on pause for a bit.

[Armand1]

Yes, I see the problem. Why not fit *all** algorithms to all pots? Using different colours, without checking them. Then let me (us) do the checking?

You see, I think it currently it goes something like this: (i) you look at pot, decide what contour algorithm to run; run one or two or more of them; (ii) check them again, perhaps binarize and smooth and check again; (iii) give the final "best" contour to me and I check them again.

How can we streamline this?

[LittleAri]

But would that really be more efficient? Checking if an algorithm has handle or not, is quite fast process. Right now, almost all of the pots use one algorithm and get two contours. If I were to test this on all algorithms then we'd have more than 10 contours per pot and the run time would be more than quintupled. I can't see how that would be efficient really when it takes a relatively short amount of time to sort the pots.

[LittleAri]

I'm happy to let you guys do the checking of whether pot contours are good and bad. So I'm happy to leave the two contours per pot image. But I don't think it's necessary to use all algorithms when I know for certain that. for example, the algorithm that finds handles for askos, is not going to work at all on bell-kraters.

[Armand1]

OK I get it. Can you, then, propose an efficient workflow along the lines that I have given? You see what I am trying to do.

[LittleAri]

Though I think you're completely right about all of this.

Currently, if you look at all algorithms and sub-algorithms, we have algorithms for:

1. pots with a handle on the right only (or no handles).
2. pots with a handle on the left only (or no handles).*
3. cups with bell-krater like handles.
4. pots (exc. cups) with bell-krater like handles but inc. cups with their handles right at the top.
5. pots with amphora-like handles.
6. pots with an amphora-like handle right at the top (e.g. askos)

* Technically 3 and 4 are very similar but after a conversation we had a while ago on the importance of cups, I made a separate algorithm for cups so I can put an even more refined range on possible handle locations and thus gather more accurate contours.

So if there are currently 6 algorithms (including sub-algorithms, and exc. the old Group 2 algorithm that I sometimes use), there'd be 12 contours for each pot.

I've also just remembered that the slowest part of the algorithm is the binarization part. So when you don't binarize the image, using all the contour extraction algorithms won't take as long as I feared. So how about this idea: I can create a new, big algorithm that has all the other algorithms within it. It will start by loading the image, _imageorig, and create a binarized version of the image, _imagebin . Then it will use all of the sub-algorithms , to find contours from _imageorig and _imagebin, and will smooth them. And the output would be 12 contour csvs and plots. Or/and one plot with 12 subplots if you'd prefer.

This way, you can send over all the pot images to me, I'll produce the 12 contours for each pot, and yourself and Han can decide which is the best one out of those. And those that don't have any contours can then be worked upon, and sent back to me to try again. If you think this would be more efficient, then I'm happy to give this a go. Advantages: No sorting needed beforehand. This will save some time on my part. Disadvantages: Many more contours for you guys to check. And run-time of algorithm would be slower (but not 5 times slower as I had first feared), but not too much, since the binarization would only happen once per image.

[Armand1]

That's along the right lines, but (I am sorry to complain), I did not realized that there were 12 algorithms and subalgorithms. Visually, that's going to be hard to sort. Can you give me fewer, for example, by running only the cup algorithms on things labelled cups? Ideally I'd have no more than, say, 4 contours to choose among. But every algorithm must have a distinctive colour so that if I sort it into the "yellow" folder, I know I need the data from the "yellow" algorithm

[LittleAri]

No need to apologise, you're completely right about all of this. At the start (many moons ago), I only had 3 algorithms, and instead of making one large one, I continued making separate ones - so it's my fault.

I could definitely merge the cup one and the other bell-krater one (3 and 4) and do as you say. That would work! I could also merge the last two since as far as I'm aware, I've only seen two classes with handles on the top and those are amphora bails and askos. So I can merge the algorithms and run no. 6 on those with names of "askos" or "bails".

So that would bring us to 4 algorithms. But I'd still like to run one with binarization and one without since although I'm trying to optimally edit images based on how light/dark they are, the methods aren't perfect yet. This would bring us to 8 contours. Or did you want me to ignore this and binarize all of them? Then there'll only be 4 contours, one for each algorithm, and I'll choose a different for each.

[Armand1]

What do we lose if we binarize everything? (and ignore non-binarized images). Are there cases where contour extraction works better on non-binarized images than binarized?

[LittleAri]

Usually the binarized ones provide much better contours but there are definitely some that are better the other way around. For example the amphora below.

The pot is quite light and parts of the background are almost the same colour. This causes the algorithm to get a bit confused between background/shadow and the actual pot when it tries to binarize the image.

[Armand1]

Let's assume, for the moment, that cases like this are rare. So, let's binarize everything.

To summarize: what I would like you to do is start with the 821 images (again). Don't sort anything. Just give me binarized images of each with 4 smoothed contours on each, distinctive colours. You have to keep track of which colour goes with which algorithm.

I will then sort them into four "colour" folders + a "bad" folder. I will then tell you how I have classified the images (into which folder they have gone) and you will return the appropriate csv to me.

We'll see where we are and how long that takes me and you.

If it is reasonable, then we have a workable pipeline for the whole thing. Bear in mind, that with 800 images, we have perhaps 5-10% of all good images that we are likely to have (20k?). So, if it takes a day to do this, then it takes only ~20 days until we have all the data for all the vases. That sounds like a lot, but it means that in a month or so we could have good contours for all of our vases. That would be a great accomplishment, and all the other stuff can come after that.

[LittleAri]

Well from the 620 or so that I had originally classes as Rank 1, I think about 80 were of non-binarized images, which is about 13%. And some classes (albeit those with very small sample sizes) had all of their good contours come from non-binarized images e.g. Lydions (probably due to the very dark strips in the bottom, middle and top). So are you sure you want to go ahead with just 4? I suppose I could always run it again for those that you find had no good contours, later on.

In that case, I will go ahead with creating one large algorithm now and merging the ones we mentioned.

[Armand1]

Let's try it. At least it's a step towards evolving an efficient procedure.

[LittleAri]

Sounds like a plan to me!

[Armand1]

I have vetted the contours that Arianna gave me on 12.02.20. These were mostly Athenian geometric, protogeometric and Lucanian RF. Arianna had already excluded 1200 "bad" contours --- these were supposed to be the good ones.

On the face of it, the results are not very encouraging.

fabric/assessment	good	bad handles	bad other
athenian geometric	549	200	173	922
athenian protogeometric	37	9	10	56
lucanian	172	61	15	248

Or, in percentages:

fabric/assessment	good	bad
athenian geometric	60	40
athenian protogeometric	66	34
lucanian	69	31

Of course we have to add the 1200 that Arianna did not give me to the bad ones. If we do, then the failure rate is around 70%.

However, I am not too discouraged. This is because athenian geometric is a rather anomalous class, and most of our vases are more like Lucanian where (I think) the success rate was higher --- I think since I don't know what Arianna held back.

There were a few big classes of failures. The first is geometric vases with little bumps for handles.

or

Lucanian pelikes also did badly

And some lucanian hydras

And lucanian nestorides

Beyond handle finding, a lot of errors are due to shadows around the base --- even when they are not obvious to the eye

And then there are miscellany of mid-contour errors caused by whitespace in the vase itself

[Armand1]

So what to do?

The quality sorted files can be seen on this dropbox.

https://www.dropbox.com/sh/vy0awgppvhhfneb/AAC45l9Gv2u39dB7ZLot3aXva?dl=0

Arianna --- can you look and assess whether your algorithms can be tweaked to fix some of these errors? The big ones are the bump handle finding. And the lucanian pelikes. We're going to have lots more of those in the future: pelikes are a big class found in Athenian BF and RF and Apulian etc. Note, you should not search for an algorithmic fix for all of them: just what is most accessible. Failing that, we are look here at fixing 1700/2400 files in photoshop. That is a lot of work -- at 30 seconds per file, about 2 days work --- but it is perfectly doable. We are in this for the long haul.

Here are some queries & comments:

1) Instead of relabelling files by fabric etc as discussed, can you just keep the file labels the same as the original? "potnumber...etc". That will assist analysis downstream. But if you can sort them into folders by fabric that would be useful.

2) instead of "flipping" contours always to the right, can you just keep them on the side you found them? That will help me assess their fit.

3) If we fix files in photoshop, can we just do it for one side instead of cutting the whole vase out? Say, consistently on the right. That will same us some time. But you'd have to know to take the contour from the right. Possible?

Looking on the bright side, we have data for 758 vases. So we are about 2-4% of the way there! The most important thing is that we now have a pipeline for data extraction. And it will only speed up.

[LittleAri]

Dear Armand, thank you for looking at these.

Good to see the Lucanians and Athenian Protogeometrics doing not-so-badly at least!

We must bear in mind this is the results from the first iteration of pot extraction. So if you recall from the plan we agreed, there will be one more iteration before we can have the finalised "good" and "bad" contours. From the 8 pot images here, 4 of them have used the incorrect algorithm e.g. the nestoris has used Algorithm 1, when in fact, it should, theoretically, work fine with Algorithm 4. Mistakes like this are due to (hopefully) be corrected in the second iteration of extraction.

You're completely right about the shadows - that's not an easy one. I'm hoping that at least a few of those that fail because of shadows, will perform better in the second iteration if removed the image binarization part of the algorithm. If not, do you think we can fix these ones by hand?

The Athenian Geometrics also enlightened me about the bug in the amphora algorithm - so they'll definitely be some amphorae that will be improved in the second round! fingers crossed

Thanks for the Dropbox link - perfect! I will do the second iteration now and then we'll have the final "good", "bad" folders, including all pots (none will be excluded in the final round). Hopefully we'll have a slightly better rate. 👍

[LittleAri]

Dear Armand,

1. Interesting - I didn't know that would be easier for you - I can certainly re-re-label them. I need the pot class which is why I relabeled them in the first place, to make things easier when it came to deciding on the algorithm. But if it's making life harder, then I'll resort back to the old names! So are we looking at a folder for each class, and then a "good" and "bad" folder within?
2. Okie dokie. Perhaps I will plot the flipped version in one colour, and the non-flipped in another colour. Would that be okay?
3. Hmmm you could do that. My only concern is the following: say you removed a very annoying handle that was on the right of an amphora, but the algorithm found a smoother outline contour on the left side of the amphora, then it will decide to use the left side, despite the fix being on the right. I can easily tweak the algorithm to always look on the left or the right, but it might cause a small number of pots to perform worse. Not too sure though.

Silver linings! 😄

[Armand1]

2. OK

3. Well, let's see how many we have to do once you've done your additional round. But I think it will be a lot so anything that we can save time with will be helpful.

Looking forward to the next round of contours.

[Armand1]

Athenian black-figure contours, after manual checking

good: 4,201 bad: 6,323 ~40% success rate. Worryingly, there are many handle-removing errors.

little master band (bad)

little master band (acceptable?}

little master band (good --- but handles were removed by me)

little master lip (bad)

skyphos (bad)

skyphos (acceptable?)

skyphos (good) --- but note lip trim, a common error

hydria (bad) common error

hydria (good)

lethykos (bad) a common error

pelike (bad)

pelike (acceptable?)

pelike (good)

stamnos (bad)

stamnos (good)

[Armand1]

The join problem

There are many vases with handles that join, for some length, directly to the body of the vase. They then blend in with body of the vase. Here is a typical example, one of thousands.

The algorithm has found the handle, removed it, but does not follow the body of the vase, rather it follows the inside of the handle where it joins to the body. Is this acceptable? I have photoshopped the handle off on the right side --- that's what I think it should be.

If that's the standard, I have to say that it will be hard for any algorithm to achieve it. This is an issue that affects thousands of amphorae.

[Armand1]

The lip problem

With some classes of pots, e.g., skyphos, there is a persistent lip finding problem. This has to be fixed

Not an image-quality problem

lekanis

[Armand1]

The kink problem

There are thousand of cups with a "kink" problem where the handle was removed. This has to be fixed

[Armand1]

The straight-chop problem

Many hydria have an excessively straight line where the handle has been removed

[Armand1]

The column krater lip problem

Column krater handles are not being accurately chopped. They're being cut flush with the inside rather than allowing for the lip. I have cut it accurately on the right.

[Armand1]

I have re-examined all the Athenian black-figure in the light of last night's conversation. I have reduced the "good" to 3,284 and "bad" to 7,236 , so really it's not a 40% success rate, but a 30% success rate.

The re-evaluation is due to several causes. (i) evaluation errors on my part: where I failed to see that the contour at the base, for example, was bad. (ii) a more stringent approach to "join", "kink", and "lip" and other contour-finding errors. Stephen really does not like them; I was (a bit out of desperation) still trying to save too many. Before I look at the vases again, these need to be fixed or else declared unfixable.

[Armand1]

I have just looked at Arianna's latest Athenian black figure test set contours. They're really much better than they were.

There was only one really bad error:

Still some difficulty here

A little here

That's not great

The krater edges remain a problem

that's not so good that's not so good

I really feel that, at this point, I should identify and clean the problem pots so that we can just move on with a good test set of contours.