search

boris-kz / CogAlg

This project is a Computer Vision implementation of general hierarchical pattern discovery principles introduced in README
http://www.cognitivealgorithm.info
MIT License
91 stars 41 forks source link

Assignment by reference may cause overwritten values from aliased copies #14

Closed Twenkid closed 4 years ago

Twenkid commented 5 years ago

https://github.com/boris-kz/CogAlg/blob/527a66491b03067cc194795d8e476ed33f417b3c/frame_dblobs.py#L146

I can't run or track it now and don't know if that's the case but that's a common source of bugs. The initial value that is correct may be overwritten by an aliased copy of the same reference by reassignments.

boris-kz commented 5 years ago

Put a breakpoint where suggested in comments.

On Thu, Oct 18, 2018, 6:52 AM Boris Kazachenko boris.kz@gmail.com wrote:

No, these are 3 separate but not exclusive error cases.

On Thu, Oct 18, 2018, 2:24 AM Todor Arnaudov notifications@github.com wrote:

IMO, such comparisons are more readable if the sequence is preserved, just the directions of <> is changed, otherwise it's more confusing/takes more time reading that it's just reversed.

if x <= _x and x - L >= ini_x: # P is fully overlapped by P if fork: # wrong: fork_ must be 0, can't be incremented before this run of while ini_x <= x: olpfork = id(fork_) # breakpoint, test prior run of while if x >= _x and x - L <= ini_x: # _P is fully overlapped by P if roots[0]: # wrong: roots mu

When both cases ==, both IF would be taken, both fork and roots. Is that the target behavior?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/boris-kz/CogAlg/issues/14#issuecomment-430889631, or mute the thread https://github.com/notifications/unsubscribe-auth/AUAXGcsGyccDVK-quK26SAr6nXNuPRdvks5umB6bgaJpZM4XQktA .

Twenkid commented 5 years ago

I did, but that's not how I could read the prior runs, they should be buffered somewhere such as in a log and then read back. Or you should record the cycles where that happened and then add a breakpoint when that cycle (number of while cycles etc.) was hit and run it again from the beginning.

boris-kz commented 5 years ago

On Thu, Oct 18, 2018 at 7:05 AM Todor Arnaudov notifications@github.com wrote:

Or you should record the cycles where that happen and then add a breakpoint when that cycle (number of while cycles etc.) it hit.

That's a good idea, maybe put: if x == error_x - error_L somewhere, then rerun, or something like that.

boris-kz commented 5 years ago

Is it correct that the first tuple of fork_ and the first tuple in a first tuple of the following list match? The first is the pattern, then the pattern again? (It's present for other non-error-test cases too)

(0, 4, 264, -46, -69, -60, -3), [((0, 4, 264, -46, -69, -60, -3, ...

Yes, this is initialized segment, 1st tuple is summed Vars, 2nd is a buffer of member Ps: Py. See comments in: if ini == 1: del(hP[:]) # blob segment [Vars, Py, ave_x, Dx, root, fork] initialized at not-included hP, replacing its fork_ refs hP += (_P[0:7]), [(_P, ave_x, 0)], ave_x, 0, [roots, [], (0,0,0,0,0,0,0,0,0)], fork

More later

boris-kz commented 5 years ago

This is how I sample buff:

if len(buff): hPt = buff[0] _Pt, _xt, _fork_t, rootst = hPt _Lt = _Pt[1] Lt = P[1] tracker.forkLog.Add(tracker.Fork("154, PRE_WHILE", _fork_t, id(_fork_t), rootst[0], id(rootst[0]), x, _xt, y, _Lt, Lt)) # time())) else: tracker.forkLog.Add(tracker.Fork("154, PRE_WHILE EMPTY BUFFER ###", fork, id(fork), [0], 0, x, -1, y, -1,P[1])) # time()))

However that's only the first one, if there are many cycles of while inix < … how many patterns are read from hP has to be counted and checked in elif hP_ I guess.

Ok, but what you need to track depends on a type of error. BTW, my third test: if buff: # wrong: buff must be [] olp_roots = id(roots) # breakpoint, test prior run of scanP is pointless because popleft changed buff. So, yours is better in that case, but it's hard to figure the context.

Twenkid commented 5 years ago

Yes, this is initialized segment, 1st tuple is summed Vars, 2nd is a buffer of member Ps: Py_. See comments in:

OK

On Thu, Oct 18, 2018 at 7:05 AM Todor Arnaudov @.***> wrote: Or you should record the cycles where that happen and then add a breakpoint when that cycle (number of while cycles etc.) it hit.

That's a good idea, maybe put: if x == error_x - error_L

but it's hard to figure the context.

Then it requires logging in the other function as well, with more parameters. The logger is and should be global anyway.

As of the location of the error, I don't get that x == ... - x changes all the time?

My suggestion was about the number of cycles of the while and scan_P calls, and also internal per pattern. Since the program runs over the same image with the same algorithm, they'd be the same.

global scanCall, counter, patternCycle
scanCall = 0; counter = 0; patternCycle = 0

def scan_P_(x, P, P_, _buff_, hP_, frame):  # P scans shared-x-coordinate _Ps in _P_, forms overlaps
    ...

    global scanCall, counter, patternCycle

    scanCall+=1
    patternCycle = 0
    if len(_buff_):
      hPt = _buff_[0]
      _Pt, _xt, _fork_t, rootst = hPt
      _Lt = _Pt[1]
      Lt = P[1]
      tracker.forkLog.Add(tracker.Fork("154, PRE_WHILE****", _fork_t, id(_fork_t), rootst[0], id(rootst[0]), x, _xt, y, _Lt, Lt, scanCall, counter, patternCycle)) # time()))
    else:
        tracker.forkLog.Add(tracker.Fork("154, PRE_WHILE**** EMPTY BUFFER ###", fork_, id(fork_), [0], 0, x, -1, y, -1,P[1], scanCall, counter, patternCycle))  # time()))
    while ini_x <= x:  # while horizontal overlap between P and hP, then P -> P_
        counter+=1
        patternCycle+=1

Then you may add a breakpoint for the precise moment of (scanCall == ), (counter == ...) and patternCycle anywhere in the program, whatever cycles of scanCall before an error that you want to investigate.

...

The first number is sequential, the line in the log, then the scan_P call number : counter that's incremented after each while(...){ ... and after / the while cycle within a scan_P call.

Below is an excerpt, the full log in fork7_exc.txt in .../cog

0: 1:0/0        154, PRE_WHILE**** EMPTY BUFFER ###     fork_id=2384457529096       fork_=0     roots=[0]       id_roots=0      x=9     _x=-1       y=405       _L=-1, L=6
1: 2:4/0        154, PRE_WHILE****      fork_id=2384453520840       fork_=0     roots=0     id_roots=1960490080     x=15        _x=4        y=405       _L=1, L=6
2: 3:9/0        154, PRE_WHILE****      fork_id=2384453520264       fork_=0     roots=1     id_roots=1960490112     x=22        _x=14       y=405       _L=5, L=7
3: 4:13/0       154, PRE_WHILE****      fork_id=2384453520008       fork_=0     roots=2     id_roots=1960490144     x=29        _x=20       y=405       _L=6, L=7
4: 5:19/0       154, PRE_WHILE****      fork_id=2384453519688       fork_=0     roots=2     id_roots=1960490144     x=34        _x=24       y=405       _L=4, L=5
5: 6:25/0       154, PRE_WHILE****      fork_id=2384453518920       fork_=0     roots=1     id_roots=1960490112     x=48        _x=33       y=405       _L=4, L=14
6: 7:29/0       154, PRE_WHILE****      fork_id=2384453562504       fork_=0     roots=2     id_roots=1960490144     x=57        _x=45       y=405       _L=12, L=9
7: 8:34/0       154, PRE_WHILE****      fork_id=2384453562824       fork_=0     roots=2     id_roots=1960490144     x=60        _x=50       y=405       _L=5, L=3
8: 9:39/0       154, PRE_WHILE****      fork_id=2384453563400       fork_=0     roots=1     id_roots=1960490112     x=64        _x=58       y=405       _L=7, L=4
9: 10:43/0      154, PRE_WHILE****      fork_id=2384453563656       fork_=0     roots=1     id_roots=1960490112     x=65        _x=62       y=405       _L=4, L=1
10: 10:45/2     178, ERROR  x <= _x and x - L >= ini_x SEE PRE_WHILE****        fork_id=2384453563912       fork_=0     roots=1     id_roots=1960490112     x=65        _x=65       y=405       _L=3, L=1
11: 11:47/0     154, PRE_WHILE****      fork_id=2384453563912       fork_=0     roots=1     id_roots=1960490112     x=72        _x=65       y=405       _L=3, L=7
12: 12:51/0     154, PRE_WHILE****      fork_id=2384453564168       fork_=0     roots=1     id_roots=1960490112     x=77        _x=66       y=405       _L=1, L=5
13: 13:55/0     154, PRE_WHILE****      fork_id=2384453564424       fork_=0     roots=1     id_roots=1960490112     x=81        _x=75       y=405       _L=9, L=4
14: 14:60/0     154, PRE_WHILE****      fork_id=2384453564680       fork_=0     roots=1     id_roots=1960490112     x=104       _x=78       y=405       _L=3, L=23
15: 14:63/3     178, ERROR  x <= _x and x - L >= ini_x SEE PRE_WHILE****        fork_id=2384453565192       fork_=0     roots=0     id_roots=1960490080     x=104       _x=104      y=405       _L=23, L=23
16: 15:65/0     154, PRE_WHILE****      fork_id=2384453565192       fork_=0     roots=1     id_roots=1960490112     x=111       _x=104      y=405       _L=23, L=7
17: 16:71/0     154, PRE_WHILE****      fork_id=2384453565448       fork_=0     roots=2     id_roots=1960490144     x=113       _x=106      y=405       _L=2, L=2
18: 16:75/4     178, ERROR  x <= _x and x - L >= ini_x SEE PRE_WHILE****        fork_id=2384453566216       fork_=0     roots=0     id_roots=1960490080     x=113       _x=113      y=405       _L=2, L=2
.....

27827: 19024:75815/2        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511369928       fork_=2     roots=1     id_roots=1960490112     x=231       _x=226      y=494       _L=8, L=13
27828: 19025:75817/0        154, PRE_WHILE****      fork_id=2384511369928       fork_=2     roots=2     id_roots=1960490144     x=234       _x=226      y=494       _L=8, L=3
27829: 19026:75821/0        154, PRE_WHILE****      fork_id=2384511370248       fork_=2     roots=2     id_roots=1960490144     x=236       _x=233      y=494       _L=7, L=2
27830: 19027:75825/0        154, PRE_WHILE****      fork_id=2384511370696       fork_=2     roots=2     id_roots=1960490144     x=242       _x=235      y=494       _L=2, L=6
27831: 19028:75829/0        154, PRE_WHILE****      fork_id=2384511371208       fork_=2     roots=2     id_roots=1960490144     x=251       _x=242      y=494       _L=7, L=9
27832: 19028:75831/2        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511371656       fork_=2     roots=1     id_roots=1960490112     x=251       _x=245      y=494       _L=3, L=9
27833: 19028:75832/3        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511372104       fork_=2     roots=1     id_roots=1960490112     x=251       _x=246      y=494       _L=1, L=9
27834: 19029:75835/0        154, PRE_WHILE****      fork_id=2384511371656       fork_=2     roots=2     id_roots=1960490144     x=257       _x=245      y=494       _L=3, L=6
27835: 19030:75841/0        154, PRE_WHILE****      fork_id=2384511373064       fork_=2     roots=1     id_roots=1960490112     x=260       _x=255      y=494       _L=5, L=3
27836: 19031:75845/0        154, PRE_WHILE****      fork_id=2384511390024       fork_=2     roots=2     id_roots=1960490144     x=264       _x=260      y=494       _L=5, L=4
27837: 19031:75847/2        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511390472       fork_=2     roots=1     id_roots=1960490112     x=264       _x=264      y=494       _L=4, L=4
27838: 19032:75849/0        154, PRE_WHILE****      fork_id=2384511390472       fork_=2     roots=2     id_roots=1960490144     x=268       _x=264      y=494       _L=4, L=4
27839: 19032:75851/2        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511390920       fork_=2     roots=1     id_roots=1960490112     x=268       _x=268      y=494       _L=4, L=4
27840: 19033:75853/0        154, PRE_WHILE****      fork_id=2384511390920       fork_=2     roots=2     id_roots=1960490144     x=274       _x=268      y=494       _L=4, L=6
27841: 19033:75855/2        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511391432       fork_=2     roots=1     id_roots=1960490112     x=274       _x=273      y=494       _L=5, L=6
27842: 19034:75857/0        154, PRE_WHILE****      fork_id=2384511391432       fork_=2     roots=2     id_roots=1960490144     x=280       _x=273      y=494       _L=5, L=6
27843: 19035:75861/0        154, PRE_WHILE****      fork_id=2384511391944       fork_=3     roots=2     id_roots=1960490144     x=285       _x=279      y=494       _L=6, L=5
27844: 19036:75865/0        154, PRE_WHILE****      fork_id=2384511392456       fork_=3     roots=2     id_roots=1960490144     x=298       _x=285      y=494       _L=6, L=13
27845: 19036:75867/2        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511393288       fork_=2     roots=1     id_roots=1960490112     x=298       _x=298      y=494       _L=13, L=13
27846: 19037:75869/0        154, PRE_WHILE****      fork_id=2384511393288       fork_=2     roots=2     id_roots=1960490144     x=303       _x=298      y=494       _L=13, L=5
27847: 19037:75871/2        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511410248       fork_=2     roots=1     id_roots=1960490112     x=303       _x=303      y=494       _L=5, L=5
27848: 19038:75873/0        154, PRE_WHILE****      fork_id=2384511410248       fork_=2     roots=2     id_roots=1960490144     x=310       _x=303      y=494       _L=5, L=7
27849: 19039:75876/0        154, PRE_WHILE****      fork_id=2384511410760       fork_=1     roots=2     id_roots=1960490144     x=313       _x=311      y=494       _L=8, L=3
27850: 19039:75878/2        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511411208       fork_=1     roots=1     id_roots=1960490112     x=313       _x=313      y=494       _L=2, L=3
27851: 19040:75880/0        154, PRE_WHILE****      fork_id=2384511410760       fork_=1     roots=2     id_roots=1960490144     x=322       _x=311      y=494       _L=8, L=9
27852: 19040:75883/3        178, ERROR  x <= _x and x - L >= ini_x SEE PRE_WHILE****        fork_id=2384511411464       fork_=3     roots=0     id_roots=1960490080     x=322       _x=322      y=494       _L=9, L=9
27853: 19041:75885/0        154, PRE_WHILE****      fork_id=2384511411464       fork_=3     roots=1     id_roots=1960490112     x=325       _x=322      y=494       _L=9, L=3
27854: 19041:75887/2        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511411912       fork_=3     roots=1     id_roots=1960490112     x=325       _x=324      y=494       _L=2, L=3
27855: 19041:75888/3        181, ERROR  x >= _x and x - L <= ini_x SEE PRE_WHILE****        fork_id=2384511412552       fork_=2     roots=1     id_roots=1960490112     x=325       _x=325      y=494       _L=1, L=3
27856: 19042:75890/0        154, PRE_WHILE****      fork_id=2384511411912       fork_=3     roots=2     id_roots=1960490144     x=326       _x=324      y=494       _L=2, L=1
boris-kz commented 5 years ago

That's a good idea, maybe put: if x == error_x - error_L

but it's hard to figure the context.

Then it requires logging in the other function as well, with more parameters. The logger is and should be global anyway.

Rerunning code with new breakpoints would do the same thing.

As of the location of the error, I don't get that x == ... - x changes all

the time?

Yes, this would be error-specific.

My suggestion was about the number of cycles of the while and scan_P calls,

and also internal per pattern. Since the program runs over the same image with the same algorithm, they'd be the same.

You can do that by specific x and y, without extra counters.

global scanCall, counter, patternCycle

scanCall = 0; counter = 0; patternCycle = 0

def scanP(x, P, P_, buff, hP_, frame): # P scans shared-x-coordinate _Ps in P, forms overlaps ...

global scanCall, counter, patternCycle

scanCall+=1
patternCycle = 0
if len(_buff_):
  hPt = _buff_[0]
  _Pt, _xt, _fork_t, rootst = hPt
  _Lt = _Pt[1]
  Lt = P[1]
  tracker.forkLog.Add(tracker.Fork("154, PRE_WHILE****", _fork_t, id(_fork_t), rootst[0], id(rootst[0]), x, _xt, y, _Lt, Lt, scanCall, counter, patternCycle)) # time()))
else:
    tracker.forkLog.Add(tracker.Fork("154, PRE_WHILE**** EMPTY BUFFER ###", fork_, id(fork_), [0], 0, x, -1, y, -1,P[1], scanCall, counter, patternCycle))  # time()))
while ini_x <= x:  # while horizontal overlap between P and hP, then P -> P_
    counter+=1
    patternCycle+=1

Then you may add a breakpoint for the precise moment of (scanCall == ), (counter == ...) and patternCycle anywhere in the program, whatever cycles of scanCall before an error that you want to investigate.

...

The first number is sequential, the line in the log, then the scan_P call number: counter that's incremented after each while(...){ ...

Below is an excerpt, the full log in fork7_exc.txt in .../cog

0: 1:0/0 154, PRE_WHILE EMPTY BUFFER ### forkid=2384457529096 fork=0 roots=[0] id_roots=0 x=9 _x=-1 y=405 _L=-1, L=6 1: 2:4/0 154, PRE_WHILE forkid=2384453520840 fork=0 roots=0 id_roots=1960490080 x=15 _x=4 y=405 _L=1, L=6 2: 3:9/0 154, PRE_WHILE forkid=2384453520264 fork=0 roots=1 id_roots=1960490112 x=22 _x=14 y=405 _L=5, L=7 3: 4:13/0 154, PRE_WHILE forkid=2384453520008 fork=0 roots=2 id_roots=1960490144 x=29 _x=20 y=405 _L=6, L=7 4: 5:19/0 154, PRE_WHILE forkid=2384453519688 fork=0 roots=2 id_roots=1960490144 x=34 _x=24 y=405 _L=4, L=5 5: 6:25/0 154, PRE_WHILE forkid=2384453518920 fork=0 roots=1 id_roots=1960490112 x=48 _x=33 y=405 _L=4, L=14 6: 7:29/0 154, PRE_WHILE forkid=2384453562504 fork=0 roots=2 id_roots=1960490144 x=57 _x=45 y=405 _L=12, L=9 7: 8:34/0 154, PRE_WHILE forkid=2384453562824 fork=0 roots=2 id_roots=1960490144 x=60 _x=50 y=405 _L=5, L=3 8: 9:39/0 154, PRE_WHILE forkid=2384453563400 fork=0 roots=1 id_roots=1960490112 x=64 _x=58 y=405 _L=7, L=4 9: 10:43/0 154, PRE_WHILE forkid=2384453563656 fork=0 roots=1 id_roots=1960490112 x=65 _x=62 y=405 _L=4, L=1 10: 10:45/2 178, ERROR x <= _x and x - L >= ini_x SEE PRE_WHILE forkid=2384453563912 fork=0 roots=1 id_roots=1960490112 x=65 _x=65 y=405 _L=3, L=1 11: 11:47/0 154, PRE_WHILE forkid=2384453563912 fork=0 roots=1 id_roots=1960490112 x=72 _x=65 y=405 _L=3, L=7 12: 12:51/0 154, PRE_WHILE forkid=2384453564168 fork=0 roots=1 id_roots=1960490112 x=77 _x=66 y=405 _L=1, L=5 13: 13:55/0 154, PRE_WHILE forkid=2384453564424 fork=0 roots=1 id_roots=1960490112 x=81 _x=75 y=405 _L=9, L=4 14: 14:60/0 154, PRE_WHILE forkid=2384453564680 fork=0 roots=1 id_roots=1960490112 x=104 _x=78 y=405 _L=3, L=23 15: 14:63/3 178, ERROR x <= _x and x - L >= ini_x SEE PRE_WHILE forkid=2384453565192 fork=0 roots=0 id_roots=1960490080 x=104 _x=104 y=405 _L=23, L=23 16: 15:65/0 154, PRE_WHILE forkid=2384453565192 fork=0 roots=1 id_roots=1960490112 x=111 _x=104 y=405 _L=23, L=7 17: 16:71/0 154, PRE_WHILE forkid=2384453565448 fork=0 roots=2 id_roots=1960490144 x=113 _x=106 y=405 _L=2, L=2 18: 16:75/4 178, ERROR x <= _x and x - L >= ini_x SEE PRE_WHILE**** forkid=2384453566216 fork=0 roots=0 id_roots=1960490080 x=113 _x=113 y=405 _L=2, L=2 .....

27827: 19024:75815/2 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511369928 fork=2 roots=1 id_roots=1960490112 x=231 _x=226 y=494 _L=8, L=13 27828: 19025:75817/0 154, PRE_WHILE forkid=2384511369928 fork=2 roots=2 id_roots=1960490144 x=234 _x=226 y=494 _L=8, L=3 27829: 19026:75821/0 154, PRE_WHILE forkid=2384511370248 fork=2 roots=2 id_roots=1960490144 x=236 _x=233 y=494 _L=7, L=2 27830: 19027:75825/0 154, PRE_WHILE forkid=2384511370696 fork=2 roots=2 id_roots=1960490144 x=242 _x=235 y=494 _L=2, L=6 27831: 19028:75829/0 154, PRE_WHILE forkid=2384511371208 fork=2 roots=2 id_roots=1960490144 x=251 _x=242 y=494 _L=7, L=9 27832: 19028:75831/2 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511371656 fork=2 roots=1 id_roots=1960490112 x=251 _x=245 y=494 _L=3, L=9 27833: 19028:75832/3 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511372104 fork=2 roots=1 id_roots=1960490112 x=251 _x=246 y=494 _L=1, L=9 27834: 19029:75835/0 154, PRE_WHILE forkid=2384511371656 fork=2 roots=2 id_roots=1960490144 x=257 _x=245 y=494 _L=3, L=6 27835: 19030:75841/0 154, PRE_WHILE forkid=2384511373064 fork=2 roots=1 id_roots=1960490112 x=260 _x=255 y=494 _L=5, L=3 27836: 19031:75845/0 154, PRE_WHILE forkid=2384511390024 fork=2 roots=2 id_roots=1960490144 x=264 _x=260 y=494 _L=5, L=4 27837: 19031:75847/2 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511390472 fork=2 roots=1 id_roots=1960490112 x=264 _x=264 y=494 _L=4, L=4 27838: 19032:75849/0 154, PRE_WHILE forkid=2384511390472 fork=2 roots=2 id_roots=1960490144 x=268 _x=264 y=494 _L=4, L=4 27839: 19032:75851/2 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511390920 fork=2 roots=1 id_roots=1960490112 x=268 _x=268 y=494 _L=4, L=4 27840: 19033:75853/0 154, PRE_WHILE forkid=2384511390920 fork=2 roots=2 id_roots=1960490144 x=274 _x=268 y=494 _L=4, L=6 27841: 19033:75855/2 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511391432 fork=2 roots=1 id_roots=1960490112 x=274 _x=273 y=494 _L=5, L=6 27842: 19034:75857/0 154, PRE_WHILE forkid=2384511391432 fork=2 roots=2 id_roots=1960490144 x=280 _x=273 y=494 _L=5, L=6 27843: 19035:75861/0 154, PRE_WHILE forkid=2384511391944 fork=3 roots=2 id_roots=1960490144 x=285 _x=279 y=494 _L=6, L=5 27844: 19036:75865/0 154, PRE_WHILE forkid=2384511392456 fork=3 roots=2 id_roots=1960490144 x=298 _x=285 y=494 _L=6, L=13 27845: 19036:75867/2 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511393288 fork=2 roots=1 id_roots=1960490112 x=298 _x=298 y=494 _L=13, L=13 27846: 19037:75869/0 154, PRE_WHILE forkid=2384511393288 fork=2 roots=2 id_roots=1960490144 x=303 _x=298 y=494 _L=13, L=5 27847: 19037:75871/2 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511410248 fork=2 roots=1 id_roots=1960490112 x=303 _x=303 y=494 _L=5, L=5 27848: 19038:75873/0 154, PRE_WHILE forkid=2384511410248 fork=2 roots=2 id_roots=1960490144 x=310 _x=303 y=494 _L=5, L=7 27849: 19039:75876/0 154, PRE_WHILE forkid=2384511410760 fork=1 roots=2 id_roots=1960490144 x=313 _x=311 y=494 _L=8, L=3 27850: 19039:75878/2 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511411208 fork=1 roots=1 id_roots=1960490112 x=313 _x=313 y=494 _L=2, L=3 27851: 19040:75880/0 154, PRE_WHILE forkid=2384511410760 fork=1 roots=2 id_roots=1960490144 x=322 _x=311 y=494 _L=8, L=9 27852: 19040:75883/3 178, ERROR x <= _x and x - L >= ini_x SEE PRE_WHILE forkid=2384511411464 fork=3 roots=0 id_roots=1960490080 x=322 _x=322 y=494 _L=9, L=9 27853: 19041:75885/0 154, PRE_WHILE forkid=2384511411464 fork=3 roots=1 id_roots=1960490112 x=325 _x=322 y=494 _L=9, L=3 27854: 19041:75887/2 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511411912 fork=3 roots=1 id_roots=1960490112 x=325 _x=324 y=494 _L=2, L=3 27855: 19041:75888/3 181, ERROR x >= _x and x - L <= ini_x SEE PRE_WHILE forkid=2384511412552 fork=2 roots=1 id_roots=1960490112 x=325 _x=325 y=494 _L=1, L=3 27856: 19042:75890/0 154, PRE_WHILE forkid=2384511411912 fork=3 roots=2 id_roots=1960490144 x=326 _x=324 y=494 _L=2, L=1

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/boris-kz/CogAlg/issues/14#issuecomment-431036053, or mute the thread https://github.com/notifications/unsubscribe-auth/AUAXGQeScRcRWok4dzvFNZqXxjOS2Iunks5umJNRgaJpZM4XQktA .

Twenkid commented 5 years ago

As of the location of the error, I don't get that x == ... - x changes all

10x, so it's 1:1 pixel to patterns.

I did a manual tracing with notes around the first ERROR, (but without watching form_P)

10: 10:45/2 178, ERROR x <= _x and x - L >= ini_x SEE PRE_WHILE**** fork_id=1879956973704 fork_=0 roots=1 id_roots=1960490112 x=65 _x=65 y=405 _L=3, L=1 _buff_ / hP contain roots == 1

It's huge for this thread, you can see the trace step by step with some notes and the values of the variables there, I guess you may understand it better : http://twenkid.com/cog/roots_tracking_x64y405.txt

The log with ERROR locations; www.twenkid.com/cog/fork7_exc.txt

BTW, you can check id's of object while debugging with the calculator icon: Evaluate expressions Alt+F8

Enter id(fork_) or whatever

boris-kz commented 5 years ago

Sorry, I haven't made any sense out of your logs yet.

Instead, I was looking at what x before error P should be... and realized that my ini_x and inix (new var) were computed wrong.

It was _x -_L and x-L, but that actually extended beyond Ps because x is included in L. Which caused scans between some disconnected Ps and _Ps.

So, I changed it to inix = x - L-1 and ini_x = _x -_L-1, last update.

That reduced number of errors: more roots == 1 and fork_ == 1 and fewer margin errors.

But there are still plenty of full overlap test errors.

And there are a lot more len(fork_) == 1 than roots[0] == 1, even though they must happen together.

And there are no roots == 1 in fork[0][4][0][0], so they must've been incremented during transfers.

So, one bug down, more to go.

BTW, Pycharm breakpoints can do a lot more than I thought, including automatic logging, conditional operations, blocking exceptions, etc.

Check it out: https://www.jetbrains.com/pycharm/features/debugger.html

On Thu, Oct 18, 2018 at 2:51 PM Todor Arnaudov notifications@github.com wrote:

As of the location of the error, I don't get that x == ... - x changes all

10x, so it's 1:1 pixel to patterns.

I did a manual tracing with notes around the first ERROR, (but without watching form_P)

10: 10:45/2 178, ERROR x <= _x and x - L >= ini_x SEE PRE_WHILE**** forkid=1879956973704 fork=0 roots=1 id_roots=1960490112 x=65 _x=65 y=405 _L=3, L=1 buff / hP contain roots == 1

It's huge, you can see the trace step by step with some notes and the values of the variables there, I guess you may understand it better : http://twenkid.com/cog/roots_tracking_x64y405.txt

The log with ERROR locations; www.twenkid.com/cog/fork7_exc.txt

BTW, you can check id's of object while debugging with the calculator icon: Evaluate expressions Alt+F8

Enter id(fork_) or whatever

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/boris-kz/CogAlg/issues/14#issuecomment-431120865, or mute the thread https://github.com/notifications/unsubscribe-auth/AUAXGSOmhotz8f4BbaE_6TEDPXOWEuNLks5umM2_gaJpZM4XQktA .

Twenkid commented 5 years ago

Sorry, I haven't made any sense out of your logs yet.

The last one was with thorough output from the debugger run step-by-step, some decoding of contents of the long tuples given from the debugger in the REPL Py to see what are the internal values and some speculations.

 _P, _x, _fork_, roots = [(1, 4, 416, 171, 149, 482, 583, [(100, 21, -40, 88, 159), (99, 57, 6, 80, 140), (103, 72, 86, 161, 118), (114, 21, 97, 153, 166)]), 62, [], [1]]
>>> _P
(1, 4, 416, 171, 149, 482, 583, [(100, 21, -40, 88, 159), (99, 57, 6, 80, 140), (103, 72, 86, 161, 118), (114, 21, 97, 153, 166)])
>>> _x
62
>>> _fork_
[]
>>> roots
[1]

p = (1, 4, 416, 171, 149, 482, 583), [((1, 4, 416, 171, 149, 482, 583, [(100, 21, -40, 88, 159), (99, 57, 6, 80, 140), (103, 72, 86, 161, 118), (114, 21, 97, 153, 166)]), 60, 0)], 60, 0, [[2], [], (0, 0, 0, 0, 0, 0, 0, 0, 0)], []
>>> import pprint
>>> pprint.pprint(p)
((1, 4, 416, 171, 149, 482, 583),
 [((1,
    4,
    416,
    171,
    149,
    482,
    583,
    [(100, 21, -40, 88, 159),
     (99, 57, 6, 80, 140),
     (103, 72, 86, 161, 118),
     (114, 21, 97, 153, 166)]),
   60,
   0)],
 60,
 0,
 [[2], [], (0, 0, 0, 0, 0, 0, 0, 0, 0)],
 [])

Instead, I was looking at what x before error P should be... and realized that my ini_x and inix (new var) were computed wrong. It was _x -_L and x-L, but that actually extended beyond Ps because x is included in L. Which caused scans between some disconnected Ps and _Ps. So, I changed it to inix = x - L-1 and ini_x = _x -_L-1, last update.

OK, then these speculations were right:

Is this image correctly drawn by the code - that's another thing that has to be verified first by checking and possibly drawing the patterns. Now you say "fork and roots should be..." but have you verified that the other variables have all correct values, too?

I couldn't know precisely (maybe besides finding a way to stop these errors) how they are supposed to be, it's "fully overlapped" either if the coordinates match completely (I noticed a lot of macthing coordinates in the ERROR) or if there's one pixel difference (from one side, from both sides...).

That has to be more precisely defined, and if there are no other specifications, the ground truth is your code.

In general +-1 border values >= <= usually are source of bugs and should be watched or precisely defined, when it's incremented, and how it's compared.

That reduced number of errors: more roots == 1 and fork == 1 and fewer margin errors. But there are still plenty of full overlap test errors. And there are a lot more len(fork) == 1 than roots[0] == 1, even though they must happen together. And there are no roots == 1 in fork[0][4][0][0], so they must've been incremented during transfers.

OK, I'll see it later.

So, one bug down, more to go.

Good.

BTW, Pycharm breakpoints can do a lot more than I thought, including automatic logging, conditional operations, blocking exceptions, etc.

Thanks. Some'd be handy. I think I may have seen it before by clicking the right button, but recently haven't tried - not suggestive GUI.

I needed to temporarily disable breakpoints or "the others", also "Run to cursor" (that's like a temporary breakpoint or remove the breakpoint once hit), but didn't find them in the menus (haven't search well).

Edit: Well, it seems it has "force run to cursor" and "temporary breakpoint" in Run menu, I haven't used the debugger a lot maybe because I wasn't sure what the values should be and what to watch.

BTW, one thing that logs (recording values, not just "printing") is better is rewinding without rerun and comparisons over many "breakpoints". For such needs even with conditional breakpoints, there'd also be a need to store these values somewhere.

boris-kz commented 5 years ago

OK, then hese speculations were right:

Is this image correctly drawn by the code - that's another thing that has to be verified first by checking and possibly drawing the patterns. Now you say "fork and roots should be..." but have you verified that the other variables have all correct values, too?

The first part is pointless, the second is a platitude. You say a lot of things, doing them is another matter.

I couldn't know precisely (maybe besides finding a way to stop these errors) how they are supposed to be, it's "fully overlapped" either if the coordinates match completely (I noticed a lot of macthing coordinates in the ERROR) or if there's one pixel difference (from one side, from both sides...).

That has to be more precisely defined, and if there are no other specifications, the ground truth is your code.

Of course you could, the meaning of vertically "fully overlapped" is obvious: all x values of one are present in x values of another. My pixel comps are vertical and lateral, not diagonal. And that's separate from determining x of initial pixel of a pattern.

Twenkid commented 5 years ago

Several times I missed to say something important, though, that I've been bored to death.

На пт, 19.10.2018 г., 15:51 Boris Kazachenko notifications@github.com написа:

OK, then hese speculations were right:

Is this image correctly drawn by the code - that's another thing that has to be verified first by checking and possibly drawing the patterns. Now you say "fork and roots should be..." but have you verified that the other variables have all correct values, too?

The first part is pointless, the second is a platitude. You say a lot of things, doing them is another matter.

I couldn't know precisely (maybe besides finding a way to stop these errors) how they are supposed to be, it's "fully overlapped" either if the coordinates match completely (I noticed a lot of macthing coordinates in the ERROR) or if there's one pixel difference (from one side, from both sides...).

That has to be more precisely defined, and if there are no other specifications, the ground truth is your code.

Of course you could, the meaning of vertically "fully overlapped" is obvious: all x values of one are present in x values of another. My pixel comps are vertical and lateral, not diagonal. And that's separate from determining x of initial pixel of a pattern.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/boris-kz/CogAlg/issues/14#issuecomment-431352494, or mute the thread https://github.com/notifications/unsubscribe-auth/AWSP2FVWHqzBvKReBG1X44cwKGB37V39ks5umcrWgaJpZM4XQktA .

Twenkid commented 5 years ago

According to my log, there are nearly 1493 cases when len(fork) is tested as one and then [0,4,0,0] is not 1 as you say it's supposed to be.

A histogram of roots value is: 2: 71 occasions (assuming "incremented by 1") 3: 762 4: 531 5: 71 6: 37

To me the requirement that the conditions you say should come together doesn't make sense or the bug is a severe mismatch.

To me it looks like using an older value of roots, one at an earlier stage than you assumed is supposed to be used.

This suggests something like that:

1262: 248:1158/0        154, PRE_WHILE****      fork_id=1916112473352       fork_=2     roots=2     id_roots=1960490144     x=178       _x=161      y=406       _L=6, L=19
1263: 248:1160/2        179, ERROR  if ini_x >= inix and _x <= x:  SEE PRE_WHILE****        fork_id=1916112473800       fork_=2     roots=1     id_roots=1960490112     x=178       _x=173      y=406       _L=12, L=19
1264: 248:1160/2        181, if s == _s         fork_id=1916112473800       fork_=2     roots=2     id_roots=1960490144     x=178       _x=173      y=406       _L=12, L=19
1265: 248:1162/4        181, if s == _s         fork_id=1916112495304       fork_=2     roots=1     id_roots=1960490112     x=178       _x=181      y=406       _L=6, L=19
1266: 249:1163/0        154, PRE_WHILE****      fork_id=1916112473352       fork_=2     roots=2     id_roots=1960490144     x=181       _x=161      y=406       _L=6, L=3
1267: 249:1164/1        181, if s == _s         fork_id=1916112473352       fork_=2     roots=3     id_roots=1960490176     x=181       _x=161      y=406       _L=6, L=3
1268: 249:1166/3        181, if s == _s         fork_id=1916112494792       fork_=1     roots=1     id_roots=1960490112     x=181       _x=175      y=406       _L=2, L=3
1269: 249:1166/3        185, if len(_fork_) == 1:       fork_id=1916112494792       fork_=1     roots=1     id_roots=1960490112     x=181       _x=175      y=406       _L=2, L=3
1270: 249:1166/3        186, if len(_fork_[0]) == ?:        fork_id=1916111749384       fork_=6     roots=1     id_roots=1960490112     x=181       _x=175      y=406       _L=2, L=3
1271: 249:1166/3        188, _fork_[0][4][0][0] == ? (roots):       fork_id=1916111749384       fork_=6     roots=3     id_roots=1960490176     x=181       _x=175      y=406       _L=2, L=3
boris-kz commented 5 years ago

Ok, I fixed a bunch bugs, just by stepping through the code. It seems to be running correctly now, although there is still a lot to check. I have 0 instances for both fork_ and roots, all functions are triggered and don't give me any errors. Error cases at 156-161 are never triggered, and no margin errors. But the code runs a lot slower now :). More latter.

On Fri, Oct 19, 2018 at 1:24 PM Todor Arnaudov notifications@github.com wrote:

According to my log, there are nearly 1493 cases when len(fork) is tested as one ant then [0,4,0,0] is not 1 as you say it's supposed to be.

A histogram of roots value is: 2: 71 occasions (assuming "incremented by 1") 3: 762 4: 531 5: 71 6: 37

To me the requirement that the conditions you say should come together doesn't make sense or the bug is a severe mismatch.

To me it looks like using an older value of roots, one at an earlier stage than you assumed is supposed to be used.

This suggests something like that:

1262: 248:1158/0 154, PRE_WHILE forkid=1916112473352 fork=2 roots=2 id_roots=1960490144 x=178 _x=161 y=406 _L=6, L=19 1263: 248:1160/2 179, ERROR if ini_x >= inix and _x <= x: SEE PRE_WHILE forkid=1916112473800 fork=2 roots=1 id_roots=1960490112 x=178 _x=173 y=406 _L=12, L=19 1264: 248:1160/2 181, if s == _s forkid=1916112473800 fork=2 roots=2 id_roots=1960490144 x=178 _x=173 y=406 _L=12, L=19 1265: 248:1162/4 181, if s == _s forkid=1916112495304 fork=2 roots=1 id_roots=1960490112 x=178 _x=181 y=406 _L=6, L=19 1266: 249:1163/0 154, PRE_WHILE**** forkid=1916112473352 fork=2 roots=2 id_roots=1960490144 x=181 _x=161 y=406 _L=6, L=3 1267: 249:1164/1 181, if s == _s forkid=1916112473352 fork=2 roots=3 id_roots=1960490176 x=181 _x=161 y=406 _L=6, L=3 1268: 249:1166/3 181, if s == _s forkid=1916112494792 fork=1 roots=1 id_roots=1960490112 x=181 _x=175 y=406 _L=2, L=3 1269: 249:1166/3 185, if len(fork) == 1: forkid=1916112494792 fork=1 roots=1 id_roots=1960490112 x=181 _x=175 y=406 _L=2, L=3 1270: 249:1166/3 186, if len(fork[0]) == ?: forkid=1916111749384 fork=6 roots=1 id_roots=1960490112 x=181 _x=175 y=406 _L=2, L=3 1271: 249:1166/3 188, fork[0][4][0][0] == ? (roots): forkid=1916111749384 fork=6 roots=3 id_roots=1960490176 x=181 _x=175 y=406 _L=2, L=3

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/boris-kz/CogAlg/issues/14#issuecomment-431436751, or mute the thread https://github.com/notifications/unsubscribe-auth/AUAXGdojVVjJ3YLwzz2VmDnOrvnMYgKSks5umgrTgaJpZM4XQktA .

Twenkid commented 5 years ago

Great, good job!

If I interpret this correctly: (the older version)

if P[0] == _P[0]:  # if s == _s: core sign match, + selective inclusion if contiguity eval?
            roots[0] += 1  # hP[3] = roots, nothing else is modified
            fork_.append(hP)  # P-connected hPs, appended with blob and converted to Py_ after P_ scan

This assignment/modification was the core of the bug: hP[3] = roots

Because roots = hP and then hP[3] = roots

So both hP and hP[3] = roots but it has to be true only for hP, because hP[3] is to be sampled in a following iteration. That also causes that value to be incremented in two runs instead of just one?

boris-kz commented 5 years ago

Thanks!

If I interpret this correctly: (the older version)

if P[0] == _P[0]: # if s == s: core sign match, + selective inclusion if contiguity eval? roots[0] += 1 # hP[3] = roots, nothing else is modified fork.append(hP) # P-connected hPs, appended with blob and converted to Py after P scan

This assignment/modification was the core of the bug: hP[3] = roots

Because roots = hP and then hP[3] = roots

So both hP and hP[3] = roots but it has to be true only for hP, because hP[3] is to be sampled in a following iteration. That also causes that value to be incremented in two runs instead of just one?

No, hP[3] = roots was superfluous because roots[0] was modified by reference. But I will put it back in because roots is going back to scalar, it is a list only for testing.

Main bugs were: