glencoesoftware / omero-ms-core

OMERO Vert.x microservice core
GNU General Public License v2.0
2 stars 6 forks source link

Handle already memoized strings and BINGET #27

Closed chris-allan closed 1 year ago

chris-allan commented 1 year ago

Under certain conditions a string may have already been "memoized" during the pickling process. If this is the case, the value during opcode sequence unwinding is handled by reference (the offset within the "memo") rather than by value.

The implementation here is a shortcut as we do not care about decoding the entire Django session but rather just the OMERO.web Connector.

See:

chris-allan commented 1 year ago

The symbolic disassembly of the new pickle (created using pickletools.dis) is as follows:

    0: \x80 PROTO      4
    2: \x95 FRAME      2161
   11: }    EMPTY_DICT
   12: \x94 MEMOIZE    (as 0)
   13: (    MARK
   14: \x8c     SHORT_BINUNICODE 'callback'
   24: \x94     MEMOIZE    (as 1)
   25: }        EMPTY_DICT
   26: \x94     MEMOIZE    (as 2)
   27: (        MARK
   28: \x8c         SHORT_BINUNICODE 'ProcessCallback/da39c719-110b-4777-bf48-a19e81f56d50 -t -e 1.1:tcp -h 10.0.13.5 -p 43649 -t 60000'
  127: \x94         MEMOIZE    (as 3)
  128: }            EMPTY_DICT
  129: \x94         MEMOIZE    (as 4)
  130: (            MARK
  131: \x8c             SHORT_BINUNICODE 'job_type'
  141: \x94             MEMOIZE    (as 5)
  142: \x8c             SHORT_BINUNICODE 'script'
  150: \x94             MEMOIZE    (as 6)
  151: \x8c             SHORT_BINUNICODE 'job_name'
  161: \x94             MEMOIZE    (as 7)
  162: \x8c             SHORT_BINUNICODE 'Export Capabilities'
  183: \x94             MEMOIZE    (as 8)
  184: \x8c             SHORT_BINUNICODE 'start_time'
  196: \x94             MEMOIZE    (as 9)
  197: \x8c             SHORT_BINUNICODE 'datetime'
  207: \x94             MEMOIZE    (as 10)
  208: \x8c             SHORT_BINUNICODE 'datetime'
  218: \x94             MEMOIZE    (as 11)
  219: \x93             STACK_GLOBAL
  220: \x94             MEMOIZE    (as 12)
  221: C                SHORT_BINBYTES b'\x07\xe7\x06\x13\x14!\x12\x02)\xe2'
  233: \x94             MEMOIZE    (as 13)
  234: \x85             TUPLE1
  235: \x94             MEMOIZE    (as 14)
  236: R                REDUCE
  237: \x94             MEMOIZE    (as 15)
  238: \x8c             SHORT_BINUNICODE 'status'
  246: \x94             MEMOIZE    (as 16)
  247: \x8c             SHORT_BINUNICODE 'finished'
  257: \x94             MEMOIZE    (as 17)
  258: \x8c             SHORT_BINUNICODE 'results'
  267: \x94             MEMOIZE    (as 18)
  268: }                EMPTY_DICT
  269: \x94             MEMOIZE    (as 19)
  270: \x8c             SHORT_BINUNICODE 'Options'
  279: \x94             MEMOIZE    (as 20)
  280: \x8c             SHORT_BINUNICODE 'TIFF PNG'
  290: \x94             MEMOIZE    (as 21)
  291: s                SETITEM
  292: u                SETITEMS   (MARK at 130)
  293: \x8c         SHORT_BINUNICODE '5bab6cda-2d78-415e-a080-958586801696/IHandle225c0aef-9c45-447a-a661-3f3cdb5da693 -t -e 1.1:tcp -h 10.0.13.5 -p 43649 -t 60000'
  420: \x94         MEMOIZE    (as 22)
  421: }            EMPTY_DICT
  422: \x94         MEMOIZE    (as 23)
  423: (            MARK
  424: \x8c             SHORT_BINUNICODE 'job_type'
  434: \x94             MEMOIZE    (as 24)
  435: \x8c             SHORT_BINUNICODE 'delete'
  443: \x94             MEMOIZE    (as 25)
  444: \x8c             SHORT_BINUNICODE 'start_time'
  456: \x94             MEMOIZE    (as 26)
  457: h                BINGET     12
  459: C                SHORT_BINBYTES b'\x07\xe7\x06\x13\x15\x04\x04\x0bQs'
  471: \x94             MEMOIZE    (as 27)
  472: \x85             TUPLE1
  473: \x94             MEMOIZE    (as 28)
  474: R                REDUCE
  475: \x94             MEMOIZE    (as 29)
  476: \x8c             SHORT_BINUNICODE 'status'
  484: \x94             MEMOIZE    (as 30)
  485: \x8c             SHORT_BINUNICODE 'finished'
  495: \x94             MEMOIZE    (as 31)
  496: \x8c             SHORT_BINUNICODE 'error'
  503: \x94             MEMOIZE    (as 32)
  504: K                BININT1    0
  506: \x8c             SHORT_BINUNICODE 'dreport'
  515: \x94             MEMOIZE    (as 33)
  516: N                NONE
  517: \x8c             SHORT_BINUNICODE 'dtype'
  524: \x94             MEMOIZE    (as 34)
  525: \x8c             SHORT_BINUNICODE 'Dataset'
  534: \x94             MEMOIZE    (as 35)
  535: \x8c             SHORT_BINUNICODE 'delmany'
  544: \x94             MEMOIZE    (as 36)
  545: \x89             NEWFALSE
  546: \x8c             SHORT_BINUNICODE 'did'
  551: \x94             MEMOIZE    (as 37)
  552: \x8c             SHORT_BINUNICODE '1'
  555: \x94             MEMOIZE    (as 38)
  556: u                SETITEMS   (MARK at 423)
  557: \x8c         SHORT_BINUNICODE '5bab6cda-2d78-415e-a080-958586801696/IHandle14ee4bc6-b003-4669-80ca-d2102fd5515f -t -e 1.1:tcp -h 10.0.13.5 -p 43649 -t 60000'
  684: \x94         MEMOIZE    (as 39)
  685: }            EMPTY_DICT
  686: \x94         MEMOIZE    (as 40)
  687: (            MARK
  688: \x8c             SHORT_BINUNICODE 'job_type'
  698: \x94             MEMOIZE    (as 41)
  699: \x8c             SHORT_BINUNICODE 'delete'
  707: \x94             MEMOIZE    (as 42)
  708: \x8c             SHORT_BINUNICODE 'start_time'
  720: \x94             MEMOIZE    (as 43)
  721: h                BINGET     12
  723: C                SHORT_BINBYTES b'\x07\xe7\x06\x13\x15\x04\t\x07\x9c8'
  735: \x94             MEMOIZE    (as 44)
  736: \x85             TUPLE1
  737: \x94             MEMOIZE    (as 45)
  738: R                REDUCE
  739: \x94             MEMOIZE    (as 46)
  740: \x8c             SHORT_BINUNICODE 'status'
  748: \x94             MEMOIZE    (as 47)
  749: \x8c             SHORT_BINUNICODE 'finished'
  759: \x94             MEMOIZE    (as 48)
  760: \x8c             SHORT_BINUNICODE 'error'
  767: \x94             MEMOIZE    (as 49)
  768: K                BININT1    0
  770: \x8c             SHORT_BINUNICODE 'dreport'
  779: \x94             MEMOIZE    (as 50)
  780: N                NONE
  781: \x8c             SHORT_BINUNICODE 'dtype'
  788: \x94             MEMOIZE    (as 51)
  789: \x8c             SHORT_BINUNICODE 'Dataset'
  798: \x94             MEMOIZE    (as 52)
  799: \x8c             SHORT_BINUNICODE 'delmany'
  808: \x94             MEMOIZE    (as 53)
  809: \x89             NEWFALSE
  810: \x8c             SHORT_BINUNICODE 'did'
  815: \x94             MEMOIZE    (as 54)
  816: \x8c             SHORT_BINUNICODE '2'
  819: \x94             MEMOIZE    (as 55)
  820: u                SETITEMS   (MARK at 687)
  821: \x8c         SHORT_BINUNICODE '5bab6cda-2d78-415e-a080-958586801696/IHandle873a5965-1576-425c-8854-b06dd4991d04 -t -e 1.1:tcp -h 10.0.13.5 -p 43649 -t 60000'
  948: \x94         MEMOIZE    (as 56)
  949: }            EMPTY_DICT
  950: \x94         MEMOIZE    (as 57)
  951: (            MARK
  952: \x8c             SHORT_BINUNICODE 'job_type'
  962: \x94             MEMOIZE    (as 58)
  963: \x8c             SHORT_BINUNICODE 'delete'
  971: \x94             MEMOIZE    (as 59)
  972: \x8c             SHORT_BINUNICODE 'start_time'
  984: \x94             MEMOIZE    (as 60)
  985: h                BINGET     12
  987: C                SHORT_BINBYTES b'\x07\xe7\x06\x13\x15\x04\x0f\x03\xcc\xfc'
  999: \x94             MEMOIZE    (as 61)
 1000: \x85             TUPLE1
 1001: \x94             MEMOIZE    (as 62)
 1002: R                REDUCE
 1003: \x94             MEMOIZE    (as 63)
 1004: \x8c             SHORT_BINUNICODE 'status'
 1012: \x94             MEMOIZE    (as 64)
 1013: \x8c             SHORT_BINUNICODE 'finished'
 1023: \x94             MEMOIZE    (as 65)
 1024: \x8c             SHORT_BINUNICODE 'error'
 1031: \x94             MEMOIZE    (as 66)
 1032: K                BININT1    0
 1034: \x8c             SHORT_BINUNICODE 'dreport'
 1043: \x94             MEMOIZE    (as 67)
 1044: N                NONE
 1045: \x8c             SHORT_BINUNICODE 'dtype'
 1052: \x94             MEMOIZE    (as 68)
 1053: \x8c             SHORT_BINUNICODE 'Dataset'
 1062: \x94             MEMOIZE    (as 69)
 1063: \x8c             SHORT_BINUNICODE 'delmany'
 1072: \x94             MEMOIZE    (as 70)
 1073: \x89             NEWFALSE
 1074: \x8c             SHORT_BINUNICODE 'did'
 1079: \x94             MEMOIZE    (as 71)
 1080: \x8c             SHORT_BINUNICODE '3'
 1083: \x94             MEMOIZE    (as 72)
 1084: u                SETITEMS   (MARK at 951)
 1085: u            SETITEMS   (MARK at 27)
 1086: \x8c     SHORT_BINUNICODE 'shares'
 1094: \x94     MEMOIZE    (as 73)
 1095: }        EMPTY_DICT
 1096: \x94     MEMOIZE    (as 74)
 1097: \x8c     SHORT_BINUNICODE 'can_create'
 1109: \x94     MEMOIZE    (as 75)
 1110: \x88     NEWTRUE
 1111: \x8c     SHORT_BINUNICODE 'active_group'
 1125: \x94     MEMOIZE    (as 76)
 1126: K        BININT1    3
 1128: \x8c     SHORT_BINUNICODE 'connector'
 1139: \x94     MEMOIZE    (as 77)
 1140: }        EMPTY_DICT
 1141: \x94     MEMOIZE    (as 78)
 1142: (        MARK
 1143: \x8c         SHORT_BINUNICODE 'server_id'
 1154: \x94         MEMOIZE    (as 79)
 1155: h            BINGET     38
 1157: \x8c         SHORT_BINUNICODE 'is_secure'
 1168: \x94         MEMOIZE    (as 80)
 1169: \x89         NEWFALSE
 1170: \x8c         SHORT_BINUNICODE 'is_public'
 1181: \x94         MEMOIZE    (as 81)
 1182: \x89         NEWFALSE
 1183: \x8c         SHORT_BINUNICODE 'omero_session_key'
 1202: \x94         MEMOIZE    (as 82)
 1203: \x8c         SHORT_BINUNICODE '29071013-a8c4-465a-9bdf-2674222ca89b'
 1241: \x94         MEMOIZE    (as 83)
 1242: \x8c         SHORT_BINUNICODE 'user_id'
 1251: \x94         MEMOIZE    (as 84)
 1252: K            BININT1    2
 1254: u            SETITEMS   (MARK at 1142)
 1255: \x8c     SHORT_BINUNICODE 'server_settings'
 1272: \x94     MEMOIZE    (as 85)
 1273: }        EMPTY_DICT
 1274: \x94     MEMOIZE    (as 86)
 1275: (        MARK
 1276: \x8c         SHORT_BINUNICODE 'ui'
 1280: \x94         MEMOIZE    (as 87)
 1281: }            EMPTY_DICT
 1282: \x94         MEMOIZE    (as 88)
 1283: (            MARK
 1284: \x8c             SHORT_BINUNICODE 'tree'
 1290: \x94             MEMOIZE    (as 89)
 1291: }                EMPTY_DICT
 1292: \x94             MEMOIZE    (as 90)
 1293: (                MARK
 1294: \x8c                 SHORT_BINUNICODE 'orphans'
 1303: \x94                 MEMOIZE    (as 91)
 1304: }                    EMPTY_DICT
 1305: \x94                 MEMOIZE    (as 92)
 1306: (                    MARK
 1307: \x8c                     SHORT_BINUNICODE 'name'
 1313: \x94                     MEMOIZE    (as 93)
 1314: \x8c                     SHORT_BINUNICODE 'Orphaned Images'
 1331: \x94                     MEMOIZE    (as 94)
 1332: \x8c                     SHORT_BINUNICODE 'enabled'
 1341: \x94                     MEMOIZE    (as 95)
 1342: \x88                     NEWTRUE
 1343: \x8c                     SHORT_BINUNICODE 'description'
 1356: \x94                     MEMOIZE    (as 96)
 1357: \x8c                     SHORT_BINUNICODE 'This is a virtual container with orphaned images. These images are not linked anywhere. Just drag them to the selected container.'
 1488: \x94                     MEMOIZE    (as 97)
 1489: u                        SETITEMS   (MARK at 1306)
 1490: \x8c                 SHORT_BINUNICODE 'type_order'
 1502: \x94                 MEMOIZE    (as 98)
 1503: \x8c                 SHORT_BINUNICODE 'tagset,tag,project,dataset,screen,plate,acquisition,image'
 1562: \x94                 MEMOIZE    (as 99)
 1563: u                    SETITEMS   (MARK at 1293)
 1564: \x8c             SHORT_BINUNICODE 'menu'
 1570: \x94             MEMOIZE    (as 100)
 1571: }                EMPTY_DICT
 1572: \x94             MEMOIZE    (as 101)
 1573: \x8c             SHORT_BINUNICODE 'dropdown'
 1583: \x94             MEMOIZE    (as 102)
 1584: }                EMPTY_DICT
 1585: \x94             MEMOIZE    (as 103)
 1586: (                MARK
 1587: \x8c                 SHORT_BINUNICODE 'colleagues'
 1599: \x94                 MEMOIZE    (as 104)
 1600: }                    EMPTY_DICT
 1601: \x94                 MEMOIZE    (as 105)
 1602: (                    MARK
 1603: \x8c                     SHORT_BINUNICODE 'label'
 1610: \x94                     MEMOIZE    (as 106)
 1611: \x8c                     SHORT_BINUNICODE 'Members'
 1620: \x94                     MEMOIZE    (as 107)
 1621: \x8c                     SHORT_BINUNICODE 'enabled'
 1630: \x94                     MEMOIZE    (as 108)
 1631: \x88                     NEWTRUE
 1632: u                        SETITEMS   (MARK at 1602)
 1633: \x8c                 SHORT_BINUNICODE 'leaders'
 1642: \x94                 MEMOIZE    (as 109)
 1643: }                    EMPTY_DICT
 1644: \x94                 MEMOIZE    (as 110)
 1645: (                    MARK
 1646: \x8c                     SHORT_BINUNICODE 'enabled'
 1655: \x94                     MEMOIZE    (as 111)
 1656: \x88                     NEWTRUE
 1657: \x8c                     SHORT_BINUNICODE 'label'
 1664: \x94                     MEMOIZE    (as 112)
 1665: \x8c                     SHORT_BINUNICODE 'Owners'
 1673: \x94                     MEMOIZE    (as 113)
 1674: u                        SETITEMS   (MARK at 1645)
 1675: \x8c                 SHORT_BINUNICODE 'everyone'
 1685: \x94                 MEMOIZE    (as 114)
 1686: }                    EMPTY_DICT
 1687: \x94                 MEMOIZE    (as 115)
 1688: (                    MARK
 1689: \x8c                     SHORT_BINUNICODE 'label'
 1696: \x94                     MEMOIZE    (as 116)
 1697: \x8c                     SHORT_BINUNICODE 'All Members'
 1710: \x94                     MEMOIZE    (as 117)
 1711: \x8c                     SHORT_BINUNICODE 'enabled'
 1720: \x94                     MEMOIZE    (as 118)
 1721: \x88                     NEWTRUE
 1722: u                        SETITEMS   (MARK at 1688)
 1723: u                    SETITEMS   (MARK at 1586)
 1724: s                SETITEM
 1725: u                SETITEMS   (MARK at 1283)
 1726: \x8c         SHORT_BINUNICODE 'browser'
 1735: \x94         MEMOIZE    (as 119)
 1736: }            EMPTY_DICT
 1737: \x94         MEMOIZE    (as 120)
 1738: \x8c         SHORT_BINUNICODE 'thumb_default_size'
 1758: \x94         MEMOIZE    (as 121)
 1759: K            BININT1    96
 1761: s            SETITEM
 1762: \x8c         SHORT_BINUNICODE 'viewer'
 1770: \x94         MEMOIZE    (as 122)
 1771: }            EMPTY_DICT
 1772: \x94         MEMOIZE    (as 123)
 1773: (            MARK
 1774: \x8c             SHORT_BINUNICODE 'initial_zoom_level'
 1794: \x94             MEMOIZE    (as 124)
 1795: K                BININT1    0
 1797: \x8c             SHORT_BINUNICODE 'roi_limit'
 1808: \x94             MEMOIZE    (as 125)
 1809: M                BININT2    2000
 1812: \x8c             SHORT_BINUNICODE 'interpolate_pixels'
 1832: \x94             MEMOIZE    (as 126)
 1833: \x88             NEWTRUE
 1834: u                SETITEMS   (MARK at 1773)
 1835: \x8c         SHORT_BINUNICODE 'download_as'
 1848: \x94         MEMOIZE    (as 127)
 1849: }            EMPTY_DICT
 1850: \x94         MEMOIZE    (as 128)
 1851: \x8c         SHORT_BINUNICODE 'max_size'
 1861: \x94         MEMOIZE    (as 129)
 1862: J            BININT     144000000
 1867: s            SETITEM
 1868: \x8c         SHORT_BINUNICODE 'web'
 1873: \x94         MEMOIZE    (as 130)
 1874: }            EMPTY_DICT
 1875: \x94         MEMOIZE    (as 131)
 1876: \x8c         SHORT_BINUNICODE 'host'
 1882: \x94         MEMOIZE    (as 132)
 1883: \x8c         SHORT_BINUNICODE ''
 1885: \x94         MEMOIZE    (as 133)
 1886: s            SETITEM
 1887: \x8c         SHORT_BINUNICODE 'scripts_to_ignore'
 1906: \x94         MEMOIZE    (as 134)
 1907: \x8c         SHORT_BINUNICODE '/omero/figure_scripts/Movie_Figure.py,/omero/figure_scripts/Split_View_Figure.py,/omero/figure_scripts/Thumbnail_Figure.py,/omero/figure_scripts/ROI_Split_Figure.py,/omero/export_scripts/Make_Movie.py,/omero/import_scripts/Populate_ROI.py'
 2147: \x94         MEMOIZE    (as 135)
 2148: \x8c         SHORT_BINUNICODE 'email'
 2155: \x94         MEMOIZE    (as 136)
 2156: \x89         NEWFALSE
 2157: u            SETITEMS   (MARK at 1275)
 2158: \x8c     SHORT_BINUNICODE 'user_id'
 2167: \x94     MEMOIZE    (as 137)
 2168: K        BININT1    2
 2170: u        SETITEMS   (MARK at 13)
 2171: .    STOP
highest protocol among opcodes = 4
chris-allan commented 1 year ago

0.7.0 release makes sense, @sbesson. I can't imagine anyone else is consuming omero-ms-core but it's good hygiene nonetheless.