Sexy Topo data table

[CaverBruce]

This may not be popular, but I am proposing that the ST data table needs a complete redesign in order for it to have functionality required for efficient and effective surveying. As a precedent, I am thinking of PocketTopo, which allows users to easily keep track of many hundreds of shots from the current survey, in conjunction with many thousands of shots from linked surveys.

Key requirements

1. Every shot must be visible to the user, in the order it was taken. Currently ST reorders all the shots according to the active station, or the station that it is moved to. This might be acceptable as a user option, but it results in utter confusion in the heat of the survey when shots do not adhere to chronological order of measurement. Related to #68
2. Every shot that comprises a survey leg must be visible to the user, rather than hiding them. In conjunction with 1. above this allows the user to make an informed decision about which shots should be included in the leg. And adjust on the fly whether 1, 3 or 5 shots are included in the averaged leg. Thsi would resolve #92
3. From and To stations on individual shots must be able to be edited. Currently ST only allows a station to be renamed. This might be OK if you want a global rename of a station for all affected shots, but I have never ever felt the need to do this, although I can envisage some use cases. What I do NEED to do regularly is to change either from station or the to station for a particular shot, and it seems ST does not permit this.
4. Undo/Redo is required for the data table (in the same way it is available for the sketch windows). A separate undo-redo stack is probably preferable for data table, plan sketch and elevation sketches.
5. Editing of the table directly in place is preferable to providing all sorts of complicated and incomplete options as is currently the case (v1.3.3). For example there would be no need for 'force new station', 'move to different station', and possibly no need for 'rename station' and 'edit row' (although that may be a way it invoke a generic editing mode). See also #87 Users would not have to guess the full implications of what those commands will do, as they could more easily edit directly in place. Compare with PocketTopo which does this exceptionally well.

Nice to have requirements TopoDroid alerts the user to shots (both when surveying and calibrating) that are suspicious due to apparent changes in earths gravitational acceleration or magnetic field strength. It would be good to have ST do this and provide some sort of beep/vibration/highlighting of shots where this occurs.

[richsmith]

OK, a lot going on here!

There's a few issues with the current table view. The UI isn't great, and largely stems from there being no editable table component in Android (no idea why, but I had to hack my own). It's also very slow due to all the hacks I had to chuck together to get it to work. There might be better options available now, a few years after I wrote it - I'll take another look some time.

I've also neglected the data view because I've tried to make most of what you want to do possible from the sketch view, to minimise jumping between views (more to do in this area).

1. The data in SexyTopo is held in a graph internally, rather than tabular, so it's just translated into a table for the data view purposes. This makes reassembling it in chronological order tricky (though possible, I'm sure, and probably should be the default or at least an option).

2. 98% of the time I don't think users are interested in what measurements make up a leg once they've generated a close-enough one. Would be nice to have a better view of them but seems like lower priority (think Qave has an expandable list, which is an interesting approach).

3. Disagree with this. The only use case I can think of for this is changing which station a measurement is from, which is better done by the move tool. In PocketTopo you can get into serious trouble if you mess this up and end up breaking your survey and creating survey "islands". I've seen this happen several times. Much better for the sake of data integrity to not allow arbitrary station changes.

4. Nice to have, but can of worms... difficult to implement and possibillity of users getting confused over the different undo stacks.

5. This might well be better (for editing measurements) but as in 1. I haven't found a good editable table library. I find I very rarely have to edit the data by hand when surveying, however.

So to summarise, I agree the UI needs to be better, but I don't think all your suggestions are ideal.

[CaverBruce]

2&3: After yesterdays survey effort, I can confirm that I need to refer to the detail of the shots that make up a survey leg (or splays that SHOULD have been a leg) about 60% of the time. Either to see if that very shaky shot had any effect on it's reading (I might want to discard it even though it passed the delta test), OR to see why the three shots have not made a leg, and delete the one that was poor, then take a third shot to hopefully trigger the smart leg algorithm. Sadly, the table design does not allow the user to unpick or mend shots, and instead requires lots of deleting of otherwise good data, and then sending the survey team back a station (if they can even find it) and retake them all. It is far quicker and easier for the booker to edit from and to stations and promote and remove splays, from, and to leg components at will, than it is to unnecessarily delete splays and shots and shoot them all again.
All of this becomes more complicated, and sometimes intractable where because the table is not chronological. With pockettopo I do this all the time, and of hundreds of surveys, there are only about four that don't plot nicely, due presumably to user error, and all of the buggy ones created perfect Therion files, so the errors were presumably not because of poor station assignment, and are in the bigger picture inconsequential. I may have seen the 'survey island' behaviour you describe, but it has always been straight forward to repair - one has to assume that bookers know what it is they are doing.

So on point 3, if ST does not move towards a fully editable data table, then I'm unlikely to be able persevere with it.

[AndrewAkinson]

All of this becomes more complicated, and sometimes intractable where because the table is not chronological. <-snip->

So on point 3, if ST does not move towards a fully editable data table, then I'm unlikely to be able persevere with it.

I am forced to agree with with Bruce, an option for chronological is critical for use and should be default for exported data (any other ordering of export should be explicitly commented, as most people assume survey data is chronological, ie if they find an instrument went wrong part way through a survey all data after that is corrupt, with ST all data has to be binned) As the note taker I cannot use data in a none chronological order, too many times have I have had data connected to the wrong station, but no idea which are the right ones and which are the wrong, accidental presses of the disto button happen quite often in squalid passages.

[richsmith]

2&3: After yesterdays survey effort, I can confirm that I need to refer to the detail of the shots that make up a survey leg (or splays that SHOULD have been a leg) about 60% of the time.

I don't relate to this - I find I or my instruments persons get the shot almost all the time (over 95% I should think - legs under 10m and making a handcage for the Disto).

So on point 3, if ST does not move towards a fully editable data table, then I'm unlikely to be able persevere with it.

You have already requested this. It's not helpful to keep asking for major changes when there's so many open issues. SexyTopo development depends on how much time I have and how I feel about it - too many requests makes me frustrated and more likely to pick up one of my other projects.

If you don't like the direction SexyTopo is going, you are free to pick up one of the other projects (although I think they have their own problems), or take a break and see if it gets better.

I thought the 4* review was a bit crappy as well given I've implemented like 20 of your suggestions.

[richsmith]

I am forced to agree with with Bruce, an option for chronological is critical for use and should be default for exported data

Yep, already agreed. On the roadmap :slightly_smiling_face:

[CaverBruce]

Sorry Rich, don't mean to pressure you, and appreciate that this is a spare time project. I don't begrudge slow progress and divergent development focus at all. It is normal in the open source world. I am just flooding with issues while they are fresh on my mind. (Apart from the name) ST is the best Android developing beta cave survey app out there. It is still just a developing beta in my opinion - I think that a rating of 4 is generous for something that is not yet ready for a real project. The ST project definitely deserves a 5 for promise, progress and responsiveness of development, but it is currently a 2 if you want to use it on a real cave survey project.

For comparison i would rate PocketTopo a 5 for concept, 4 for its current usability, and 1 for recent development.
Topodroid, a 3 for concept, 2 for usability (sorry Marco, its just not efficient in a cave and exporting to Therion) and a 5 for responsiveness of development and some great new initiatives like gravitational and acceleration checks.

In my world not many apps get a 5. You still have my backing, if you did not I would not be raising the issues, I have seen no other viable cave apps! Is there a ST discussion forum or wiki? I think it is time.

[CaverBruce]

... I can confirm that I need to refer to the detail of the shots that make up a survey leg (or splays that SHOULD have been a leg) about 60% of the time.

... ... I don't relate to this - I find I or my instruments persons get the shot almost all the time (over 95% I should think - legs under 10m and making a handcage for the Disto).

Agree that shots are good more than 95% of time, especially with two handed disto usage. But I have reason or curiosity to check 60% of the time. Especially as i know(?) that ST does not provide recourse to converting legs to splays or splays to (aggregated) legs. This leads to data integrity anxiety, and the need to check frequently. Too often I find my fears are well founded, and I cannot make the tweaks I am used to making, requiring otherwise good shots to be deleted and retaken.

[richsmith]

I appreciate the ideas and testing, but a bit of psychology is useful. 4* feels critical in the world of online reviews. Might be better not to. It's not like a restaurant review...

Digression on SexyTopo v TopoDroid v PocketTopo - PocketTopo is excellent but closed and source and runs on old hardware. I've got more impressed after implementing some of the same features. It's an amazing bit of work.

I'm less keen on TopoDroid [criticisms deleted, because I don't want to diss other caving free software projects.]

You're towards the expert/power user end of cave surveyor I would think. I have about 250 installs, so probably a lot more users who use it. For most of them they probably learn to survey on the job on an expedition. They need sensible defaults, and an easy to use interface. I think SexyTopo probably is the best surveying option for most people. It's been/being used seriously on major projects (including by me).

[richsmith]

I think I've got chronological data export / tables mostly sorted. Needs some more testing.

[CaverBruce]

FYI I just carried out a survey (with PocketTopo) that I don’t think could have been booked with SexyTopo (and possibly TopoDroid – but not familiar enough to be sure). Just confirming to myself that the user needs to have transparent display of all shots that comprise legs, and full editing control of from and to stations assigned to a particular leg. PocketTopo achieves this with clarity, and with no need for complicated user dialog logic and coding.

Scenario was a shaft system with a light water mist filling the space that was not actually in the water fall zone. Individual pitches about 20 to 40 m. Not large but enough to make communication difficult (both verbal and Bluetooth). In short, no clear comms until both disto and booker were able to congregate at the top of pitch series. Similar surveying conundrums occur in difficult tight wet muddy rift passages. There is only one opportunity to traverse the passage as no sane team member will stay on board if the booker says the software will make me delete the last three shots just to fix a very minor disto error and we all have to backup and do it again despite hypothermia. As happens with all of the four distos I have used, in adverse moisture or target conditions, the assumed ‘real’ distance of say 22 m is regularly reported as 0.2m (disto case length), 50m , 160m or more. Credible and repeatable distances are almost always evident, but sometimes in difficult conditions two or more good readings might only be obtained in 10 or more shots. ie we might never get three consecutive good shots. With no communication the disto person tends to take many shots, as they work their way up the pitch, and then once together with the booker, the shots are weeded to leave only credible data from which to craft legs and splays. This understandably results in some legs being duplicated one or two times, requiring leg deletions and leg renumbering. No problem in PocketTopo, just delete and manually renumber using the + and – buttons. Survey ‘islands’ occur if you make a mistake, but if you check the graphic views these are easy to detect and sort out before continuing the survey. I tried this with some SexyTopo data, and deleting a leg just deletes the leg and all of the trailing shots (splays and legs) to the end of the current branch. Absolutely not what the user would hope for in the above scenario. And why delete all the splays along with the leg? In the case of an unintended extra leg, the splays are most likely perfectly good data. There is a warning dialog, that says x and y number of legs and splays will be deleted. What it should say is that all trailing data will be deleted. And changing the from or to station on a leg is not possible… So I come back to the conclusion that SexyTopo is only useful in benign situations where disto errors are never made, or where there is no difficulty deleting a chunk of good data just because a single shot is misfired 10 shots back.

So I’m still holding out until the data table is freely editable. A benefit is also that the coder does not have to second guess all likely use cases.