search

GemTalk / Jadeite

IDE for GemStone Smalltalk application development using Rowan code management
MIT License
6 stars 2 forks source link

Handle unreadable characters gracefully #453

Closed ericwinger closed 1 year ago

ericwinger commented 5 years ago

When doing a changes command from the changes browser, the customer had a unicode character in the comment field. Until unicode is fully supported by Jadeite in #307, we need to handle this more gracefully. Attaching a stack and customer-generated screen shots. Looks like the problem is in our STON implementation. 

***** Starting Dolphin stack dump at 08:39:34 25.02.2019 *****

************************** Dolphin Virtual Machine Dump Report ***************************

08:39:34, 25.02.2019: Can't hold $\x2022

*----> VM Context <----*
Process: {21330210:size 694 words, suspended frame 21330C85, priority 6, callbacks 1
last failure 2:nil, FP control 80017, thread 00000000}
Active Method: SessionManager>>logError:
IP: 003EE5A9 (9)
SP: 21330DEC
BP: 21330DC0 (733)
ActiveFrame: {21330DC4: cf 21330DA5, sp 21330DDC, bp 21330DC0, ip 5, JadeSessionManager(SessionManager)>>logError:}
    receiver: a JadeSessionManager
    arg[0]: a Error

New Method: VMLibrary>>dump:path:stackDepth:walkbackDepth:
Message Selector: dump:path:stackDepth:walkbackDepth:

*----> Stack Back Trace <----*
{21330DC4: cf 21330DA5, sp 21330DDC, bp 21330DC0, ip 5, JadeSessionManager(SessionManager)>>logError:}
    receiver: a JadeSessionManager
    arg[0]: a Error

{21330DA4: cf 21330D81, sp 21330DB8, bp 21330D9C, ip 39, JadeSessionManager>>logError:}
    receiver: a JadeSessionManager
    arg[0]: a Error
    stack temp[0]: '08:39:34 25.02.2019'
    env temp[0]: a FileStream

{21330D80: cf 21330D61, sp 21330D94, bp 21330D7C, ip 3, JadeSessionManager(SessionManager)>>unhandledException:}
    receiver: a JadeSessionManager
    arg[0]: a Error

{21330D60: cf 21330D41, sp 21330D74, bp 21330D5C, ip 3, JadeSessionManager(SessionManager)>>onUnhandledError:}
    receiver: a JadeSessionManager
    arg[0]: a Error

{21330D40: cf 21330D25, sp 21330D54, bp 21330D40, ip 5, Error>>defaultAction}
    receiver: a Error

{21330D24: cf 21330CF1, sp 21330D38, bp 21330D0C, ip 55, Error(Exception)>>_propagateFrom:}
    receiver: a Error
    arg[0]: a ExceptionHandler
    stack temp[0]: nil
    stack temp[1]: a ExceptionHandler
    stack temp[2]: nil
    stack temp[3]: a Process('Main' base 21330000 [ACTIVE] in SessionManager>>logError: sp=00000000 ip=4 list=024A0010)
    stack temp[4]: nil

{21330CF0: cf 21330CD1, sp 21330D04, bp 21330CEC, ip 6, Error(Exception)>>_propagate}
    receiver: a Error
    stack temp[0]: nil

{21330CD0: cf 21330CB5, sp 21330CE4, bp 21330CD0, ip 13, Error(Exception)>>signal}
    receiver: a Error

{21330CB4: cf 21330C95, sp 21330CC8, bp 21330CB0, ip 5, Error(Exception)>>signal:}
    receiver: a Error
    arg[0]: 'Can't hold $\x2022'

{21330C94: cf 21330C75, sp 21330CA8, bp 21330C90, ip 5, Error class(Exception class)>>signal:}
    receiver: Error
    arg[0]: 'Can't hold $\x2022'

{21330C74: cf 21330C55, sp 21330C88, bp 21330C70, ip 4, String(Object)>>error:}
    receiver: '1403288321\09\closest ancestor vs. loaded\0A\M\09\1403288065\09\M\09\dkh 3/29/2018 14:21\09\DBOUnb...'
    arg[0]: 'Can't hold $\x2022'

{21330C54: cf 21330C35, sp 21330C68, bp 21330C50, ip 6, String(Object)>>errorCantHold:}
    receiver: '1403288321\09\closest ancestor vs. loaded\0A\M\09\1403288065\09\M\09\dkh 3/29/2018 14:21\09\DBOUnb...'
    arg[0]: $\x2022

{21330C34: cf 21330C11, sp 21330C48, bp 21330C2C, ip 24, String(Object)>>errorAt:put:}
    receiver: '1403288321\09\closest ancestor vs. loaded\0A\M\09\1403288065\09\M\09\dkh 3/29/2018 14:21\09\DBOUnb...'
    arg[0]: 3138
    arg[1]: $\x2022

{21330C10: cf 21330BED, sp 21330C24, bp 21330C08, ip 8, String>>at:put:}
    receiver: '1403288321\09\closest ancestor vs. loaded\0A\M\09\1403288065\09\M\09\dkh 3/29/2018 14:21\09\DBOUnb...'
    arg[0]: 3138
    arg[1]: $\x2022

{21330BEC: cf 21330BCD, sp 21330C00, bp 21330BE8, ip 15, WriteStream>>nextPut:}
    receiver: a WriteStream
    arg[0]: $\x2022

{21330BCC: cf 21330BA9, sp 21330BE0, bp 21330BC4, ip 41, [] in STONReader>>parseStringInternal}
    receiver: a STONReader
    arg[0]: a WriteStream

{21330BA8: cf 21330B89, sp 21330BBC, bp 21330BA4, ip 18, STONReader>>stringStreamContents:}
    receiver: a STONReader
    arg[0]: [] @ 27 in STONReader>>parseStringInternal

{21330B88: cf 21330B65, sp 21330B9C, bp 21330B80, ip 47, STONReader>>parseStringInternal}
    receiver: a STONReader
    stack temp[0]: nil
    stack temp[1]: $'

{21330B64: cf 21330B49, sp 21330B78, bp 21330B64, ip 2, STONReader>>parseString}
    receiver: a STONReader

{21330B48: cf 21330B25, sp 21330B5C, bp 21330B44, ip 42, STONReader>>parseValue}
    receiver: a STONReader
    stack temp[0]: $'

{21330B24: cf 21330B01, sp 21330B3C, bp 21330B1C, ip 20, STONReader>>parseListDo:}
    receiver: a STONReader
    arg[0]: [] @ 20 in STONReader>>parseList
    stack temp[0]: 4

{21330B00: cf 21330AE1, sp 21330B14, bp 21330AFC, ip 25, [] in STONReader>>parseList}
    receiver: a STONReader
    arg[0]: a WriteStream

{21330AE0: cf 21330AB9, sp 21330AF4, bp 21330AD4, ip 6, Array class(SequenceableCollection class)>>new:streamContents:}
    receiver: Array
    arg[0]: 100
    arg[1]: [] @ 12 in STONReader>>parseList
    stack temp[0]: a WriteStream

{21330AB8: cf 21330A99, sp 21330ACC, bp 21330AB4, ip 6, Array class(SequenceableCollection class)>>streamContents:}
    receiver: Array
    arg[0]: [] @ 12 in STONReader>>parseList

{21330A98: cf 21330A75, sp 21330AAC, bp 21330A90, ip 27, STONReader>>parseList}
    receiver: a STONReader
    stack temp[0]: a STONReference
    stack temp[1]: nil

{21330A74: cf 21330A55, sp 21330A88, bp 21330A70, ip 28, STONReader>>parseValue}
    receiver: a STONReader
    stack temp[0]: $[

{21330A54: cf 21330A2D, sp 21330A68, bp 21330A48, ip 40, STONReader>>parseMapDo:}
    receiver: a STONReader
    arg[0]: [] @ 25 in Object>>fromSton:
    stack temp[0]: changes
    stack temp[1]: a Array

{21330A2C: cf 21330A05, sp 21330A40, bp 21330A24, ip 33, RowanProjectService(Object)>>fromSton:}
    receiver: a RowanProjectService
    arg[0]: a STONReader
    stack temp[0]: a Array

{21330A04: cf 213309E5, sp 21330A1C, bp 21330A00, ip 6, RowanProjectService class(Object class)>>fromSton:}
    receiver: RowanProjectService
    arg[0]: a STONReader

{213309E4: cf 213309BD, sp 213309F8, bp 213309D8, ip 8, STONReader>>parseObject}
    receiver: a STONReader
    stack temp[0]: RowanProjectService
    stack temp[1]: a STONReference
    stack temp[2]: nil

{213309BC: cf 21330999, sp 213309D0, bp 213309B8, ip 12, STONReader>>parseValue}
    receiver: a STONReader
    stack temp[0]: $R

{21330998: cf 21330975, sp 213309B0, bp 21330990, ip 20, STONReader>>parseListDo:}
    receiver: a STONReader
    arg[0]: [] @ 20 in STONReader>>parseList
    stack temp[0]: 1

{21330974: cf 21330955, sp 21330988, bp 21330970, ip 25, [] in STONReader>>parseList}
    receiver: a STONReader
    arg[0]: a WriteStream

{21330954: cf 2133092D, sp 21330968, bp 21330948, ip 6, Array class(SequenceableCollection class)>>new:streamContents:}
    receiver: Array
    arg[0]: 100
    arg[1]: [] @ 12 in STONReader>>parseList
    stack temp[0]: a WriteStream

{2133092C: cf 2133090D, sp 21330940, bp 21330928, ip 6, Array class(SequenceableCollection class)>>streamContents:}
    receiver: Array
    arg[0]: [] @ 12 in STONReader>>parseList

{2133090C: cf 213308E9, sp 21330920, bp 21330904, ip 27, STONReader>>parseList}
    receiver: a STONReader
    stack temp[0]: a STONReference
    stack temp[1]: nil

{213308E8: cf 213308C9, sp 213308FC, bp 213308E4, ip 28, STONReader>>parseValue}
    receiver: a STONReader
    stack temp[0]: $[

{213308C8: cf 213308A9, sp 213308DC, bp 213308C4, ip 4, STONReader>>next}
    receiver: a STONReader
    stack temp[0]: nil

{213308A8: cf 21330889, sp 213308BC, bp 213308A4, ip 5, STON class>>fromStream:}
    receiver: STON
    arg[0]: a ReadStream

{21330888: cf 21330869, sp 2133089C, bp 21330884, ip 13, STON class>>fromString:}
    receiver: STON
    arg[0]: '[RowanProjectService{#name:'DBOMain',#sha:'4f01ca8',#branch:'develop',#isSkew:tr...'

{21330868: cf 21330841, sp 2133087C, bp 2133085C, ip 6, BrowserUpdate>>update:afterStonReplication:}
    receiver: a BrowserUpdate
    arg[0]: a Array
    arg[1]: '[RowanProjectService{#name:'DBOMain',#sha:'4f01ca8',#branch:'develop',#isSkew:tr...'
    stack temp[0]: nil

{21330840: cf 21330819, sp 21330854, bp 21330834, ip 50, BrowserUpdate>>basicIssueCommand:session:}
    receiver: a BrowserUpdate
    arg[0]: a Array
    arg[1]: a GciSession
    stack temp[0]: '[RowanProjectService{#name:'DBOMain',#isDirty:false,#command:#changes,#sha:'4f01...'
    env temp[0]: '[RowanProjectService{#name:'DBOMain',#sha:'4f01ca8',#branch:'develop',#isSkew:tr...'

{21330818: cf 213307ED, sp 2133082C, bp 21330810, ip 27, [] in BrowserUpdate>>issueCommand:session:}
    receiver: a BrowserUpdate

{213307EC: cf 213307CD, sp 21330800, bp 213307E8, ip 18, BlockClosure>>ifCurtailed:}
    receiver: [] @ 0 in nil
    arg[0]: [] @ 35 in BrowserUpdate>>issueCommand:session:

{213307CC: cf 213307A9, sp 213307E0, bp 213307C4, ip 3, BlockClosure>>ensure:}
    receiver: [] @ 24 in BrowserUpdate>>issueCommand:session:
    arg[0]: [] @ 35 in BrowserUpdate>>issueCommand:session:
    stack temp[0]: nil

{213307A8: cf 2133077D, sp 213307BC, bp 213307A0, ip 40, [] in BrowserUpdate>>issueCommand:session:}
    receiver: a BrowserUpdate

{2133077C: cf 2133075D, sp 21330790, bp 21330778, ip 18, BlockClosure>>ifCurtailed:}
    receiver: [] @ 0 in nil
    arg[0]: [] @ 48 in BrowserUpdate>>issueCommand:session:

{2133075C: cf 21330739, sp 21330770, bp 21330754, ip 53, BrowserUpdate>>issueCommand:session:}
    receiver: a BrowserUpdate
    arg[0]: a Array
    arg[1]: a GciSession

{21330738: cf 21330715, sp 2133074C, bp 21330730, ip 6, JadeiteProjectListPresenter class(JadePresenter class)>>issueCommand:session:}
    receiver: JadeiteProjectListPresenter
    arg[0]: a Array
    arg[1]: a GciSession

{21330714: cf 213306F5, sp 21330728, bp 21330710, ip 6, JadeiteProjectListPresenter(JadePresenter)>>issueCommand:}
    receiver: a JadeiteProjectListPresenter
    arg[0]: a Array

{213306F4: cf 213306CD, sp 21330708, bp 213306E8, ip 8, RowanProjectService>>changesUsing:}
    receiver: a RowanProjectService
    arg[0]: a JadeiteProjectListPresenter
    stack temp[0]: nil
    stack temp[1]: nil

{213306CC: cf 213306B1, sp 213306E0, bp 213306CC, ip 14, JadeiteProjectListPresenter>>projectChanges}
    receiver: a JadeiteProjectListPresenter

{213306B0: cf 21330691, sp 213306C4, bp 213306AC, ip 4, Symbol>>forwardTo:}
    receiver: projectChanges
    arg[0]: a JadeiteProjectListPresenter

{21330690: cf 21330671, sp 213306A4, bp 2133068C, ip 4, CommandDescription>>performAgainst:}
    receiver: a CommandDescription
    arg[0]: a JadeiteProjectListPresenter

{21330670: cf 2133064D, sp 21330684, bp 21330670, ip 14, [] in Command>>value}
    receiver: a Command

{2133064C: cf 2133062D, sp 21330660, bp 21330648, ip 18, BlockClosure>>ifCurtailed:}
    receiver: [] @ 0 in nil
    arg[0]: [] @ 17 in Command>>value

{2133062C: cf 21330609, sp 21330640, bp 21330624, ip 3, BlockClosure>>ensure:}
    receiver: [] @ 11 in Command>>value
    arg[0]: [] @ 17 in Command>>value
    stack temp[0]: nil

{21330608: cf 213305ED, sp 2133061C, bp 21330608, ip 22, Command>>value}
    receiver: a Command

{213305EC: cf 213305CD, sp 21330600, bp 213305E8, ip 30, ShellView>>performCommand:}
    receiver: a ShellView
    arg[0]: a Command

{213305CC: cf 213305AD, sp 213305E0, bp 213305C8, ip 4, JadeiteTranscript(Shell)>>performCommand:}
    receiver: a JadeiteTranscript
    arg[0]: a Command

{213305AC: cf 21330585, sp 213305C0, bp 213305A0, ip 10, CommandQuery>>perform}
    receiver: a CommandQuery
    stack temp[0]: a Command
    stack temp[1]: nil
    stack temp[2]: a JadeiteTranscript

{21330584: cf 21330561, sp 21330598, bp 2133057C, ip 8, DelegatingCommandPolicy(CommandPolicy)>>route:}
    receiver: a DelegatingCommandPolicy
    arg[0]: a CommandDescription
    stack temp[0]: a CommandQuery

{21330560: cf 21330539, sp 21330574, bp 2133055C, ip 15, [] in ListView(View)>>onCommand:}
    receiver: a ListView

{21330538: cf 21330519, sp 2133054C, bp 21330534, ip 18, BlockClosure>>ifCurtailed:}
    receiver: [] @ 0 in nil
    arg[0]: [] @ 18 in Cursor>>showWhile:

{21330518: cf 213304F5, sp 2133052C, bp 21330510, ip 3, BlockClosure>>ensure:}
    receiver: [] @ 11 in View>>onCommand:
    arg[0]: [] @ 18 in Cursor>>showWhile:
    stack temp[0]: nil

{213304F4: cf 213304CD, sp 21330508, bp 213304E8, ip 27, Cursor>>showWhile:}
    receiver: a Cursor
    arg[0]: [] @ 11 in View>>onCommand:
    stack temp[0]: nil
    stack temp[1]: a ExternalHandle(00010005)

{213304CC: cf 213304A9, sp 213304E0, bp 213304C4, ip 17, ListView(View)>>onCommand:}
    receiver: a ListView
    arg[0]: a CommandDescription
    stack temp[0]: nil

{213304A8: cf 21330475, sp 213304BC, bp 21330490, ip 19, ListView(View)>>wmCommand:wParam:lParam:}
    receiver: a ListView
    arg[0]: 273
    arg[1]: 6526
    arg[2]: 0
    stack temp[0]: 6526
    stack temp[1]: a CommandDescription
    stack temp[2]: nil

{21330474: cf 21330445, sp 21330488, bp 21330460, ip 23, ListView(View)>>dispatchMessage:wParam:lParam:}
    receiver: a ListView
    arg[0]: 273
    arg[1]: 6526
    arg[2]: 0
    stack temp[0]: nil
    stack temp[1]: wmCommand:wParam:lParam:

{21330444: cf 21330411, sp 21330458, bp 21330434, ip 69, [] in InputState>>wndProc:message:wParam:lParam:cookie:}
    receiver: nil

{21330410: cf 213303E9, sp 21330424, bp 2133040C, ip 18, BlockClosure>>ifCurtailed:}
    receiver: [] @ 0 in nil
    arg[0]: [] @ 79 in InputState>>wndProc:message:wParam:lParam:cookie:

{213303E8: cf 213303AD, sp 21330404, bp 213303D0, ip 84, InputState>>wndProc:message:wParam:lParam:cookie:}
    receiver: a InputState
    arg[0]: 3936204
    arg[1]: 273
    arg[2]: 6526
    arg[3]: 0
    arg[4]: 818586
    stack temp[0]: a ListView

{213303AC: cf 2133038D, sp 213303C8, bp 213303A8, ip 28, InputState>>pumpMessage:}
    receiver: a InputState
    arg[0]: a MSG

{2133038C: cf 21330361, sp 213303A0, bp 2133037C, ip 18, InputState>>loopWhile:}
    receiver: a InputState
    arg[0]: [] @ 9 in InputState>>mainLoop
    stack temp[0]: a MSG
    stack temp[1]: true
    stack temp[2]: nil

{21330360: cf 21330345, sp 21330374, bp 21330360, ip 15, InputState>>mainLoop}
    receiver: a InputState

{21330344: cf 21330329, sp 21330358, bp 21330344, ip 14, [] in InputState>>forkMain}
    receiver: a InputState

{21330328: cf 2133030D, sp 2133033C, bp 21330328, ip 13, ExceptionHandler(ExceptionHandlerAbstract)>>markAndTry}
    receiver: a ExceptionHandler

{2133030C: cf 213302E5, sp 21330320, bp 21330308, ip 22, [] in ExceptionHandler(ExceptionHandlerAbstract)>>try:}
    receiver: a ExceptionHandler

{213302E4: cf 213302C5, sp 213302F8, bp 213302E0, ip 18, BlockClosure>>ifCurtailed:}
    receiver: [] @ 0 in nil
    arg[0]: [] @ 39 in ExceptionHandlerAbstract>>try:

{213302C4: cf 213302A1, sp 213302D8, bp 213302BC, ip 3, BlockClosure>>ensure:}
    receiver: [] @ 17 in ExceptionHandlerAbstract>>try:
    arg[0]: [] @ 39 in ExceptionHandlerAbstract>>try:
    stack temp[0]: nil

{213302A0: cf 21330279, sp 213302B4, bp 21330294, ip 44, ExceptionHandler(ExceptionHandlerAbstract)>>try:}
    receiver: a ExceptionHandler
    arg[0]: [] @ 10 in InputState>>forkMain
    stack temp[0]: nil
    stack temp[1]: a Process('Main' base 21330000 [ACTIVE] in SessionManager>>logError: sp=00000000 ip=4 list=024A0010)
    env temp[0]: nil

{21330278: cf 21330255, sp 2133028C, bp 21330270, ip 7, BlockClosure>>on:do:}
    receiver: [] @ 10 in InputState>>forkMain
    arg[0]: ProcessTermination
    arg[1]: [] @ 13 in BlockClosure>>newProcess

{21330254: cf 00000001, sp 21330268, bp 21330254, ip 17, [] in BlockClosure>>newProcess}
    receiver: [] @ 10 in InputState>>forkMain

<Bottom of stack>

***** End of dump *****

***** Ending Dolphin stack dump started at 08:39:34 25.02.2019 *****

From customer:

When there are some mildly exotic characters in the source code (maybe just in comments), it causes "Changes" between GIT and Gemstone to fail. The character is for example a bullet point ("•" - Unicode 0x2022).

Customer screen shots: image image image

ericwinger commented 4 years ago

Dolphin 7.1.9 supports unicode characters better. Here is what happens now in Jadeite.

Change (Added bullet point to method): image

Changes browser now shows: image

Going to assign to @LisaAlmarode for further testing in the ported code.

LisaAlmarode commented 4 years ago

This issue is confusing since a bullet is not always a bullet, and it can be hard to know what you are really working with using a GUI.

If you paste a bullet into the UI, it's confusing since dophin sort of supports the 1252 code page which re-uses ascii 149 for unicode 2020. This would have worked in the older dolphin codebase and is presumably what the customer is using and seeing?

image

image

Note that an actual bullet with unicode 2022 (decimal 8226), can't be pasted in, but code can be generated using:

RowanSample4Test rwCompileMethod:'test5
" comment ', 
(String with: (Character withValue: 8226)),
' end comment"
^4'
 category: 'tests'

This gets warnings everywhere, which is correct, and appears to be handled correctly.

image

It's not entirely clear what the customer is doing or what Eric is doing...

ericwinger commented 4 years ago

@LisaAlmarode The entire issue needs to be visited thoroughly, especially STON replication of unicode strings. However, for now, is the issue to make the Changes Browser handle the bullet?

dalehenrich commented 4 years ago

... it make sense to handle the Unicode characters using RSR ... I think Kurt has tests cases for passing Unicode characters/strings around, so he might be solving the problem in Dolphin as well ... I can check with Kurt if you want...

LisaAlmarode commented 4 years ago

@ericwinger's comments, I believe, are related to bullet 149, the codepage 1252 bullet. not the real unicode bullet 2022. Handling of bullet 149 and other code page characters is really bad now and that should be looked at.

I think we've cleaned up unicode strings from rowan pretty well and that may not be an issue any more. The customer appears to have actual bullet 2022 in their code.

LisaAlmarode commented 1 year ago

Changes seems to be looking okay as of 3.2.12