Unit tests fail on big-endian architectures

matoro commented 1 year ago

Hi, the unit tests do not pass on big-endian architectures (6 pass, 74 fail). Here is what the output looks like (diffed against Testing/exp/example_data.128.1.len.out_EXP):

T11FA13B57F78913B236924271569EA6D1FB2C451C33668484552C812D33138B8C73FFCE        example_data/021106_yossivassa.txt
T145D14807A78523B35A030267671FA2C2F725402973629B25545EB43C3356679477F7FC        example_data/0Alice.txt
T1C8E18E277B98A6255FA70063B70FBAE1F6618229332C9438084FB331178ADEE53B41D5        example_data/11-17-06_Academy.txt
T1E951874702042376169012B1BA5A76EAF36092FC3311A595B4856235278F9F973763EF        example_data/1english-only.txt
T106D25917F780237185070293B60E36FAB735C0F833D66460688DA22D6756E751B7BAEB        example_data/2005NISSE.txt
T135210431F3C6AEA059EB1ABA870E8C4758D70024C63224B5D4C89354E46315D26BF736        example_data/2006-07_Resource_Brochure.txt
T1C0810B63DB6455D3411303561428A9FBE1369B047760286686FFA067B2DA70D823DEFF        example_data/2006_2007PhysicalEducationConceptMap.txt
T1E2226B67EB006374230721F2620F5283EF3E817A6265462A745DF36933E7690037F7AA        example_data/2007ShowcaseFilm_Package.txt
T17E525967B7C413B61A93119266DD11E2E37A49BCB3718C61B04E8438235ECB7477B3EA        example_data/22-ppa-3rd_e2snewsletter_jun06.txt
T14221159BD71126276E6282B9668A6A81FF08C5293735431AA0CE50002B17DAE67FA7D4        example_data/42nd_street.txt
T1F5027E27F36923366E020362E50EA1F1D76943BC935A4221454DE03C3276D6E5BBF6ED        example_data/5._.txt
T17D512452EBC42331ABD704331B5E5786D319172DD3064C22BD4D82A42B864A91B3B7EC        example_data/7-prikaz-slozeno.txt
T1E9E17BBDEB0873EB275983DEB84150A09731C51453904A40391D7B3A135ABBA4FB7E79        example_data/AccessToNationalSportsCentresPassportScheme.txt
T1EF71381FBB7D13058AC3A3956318C2D9BBBCC25DF3B5499132DE6118377A12D433A654        example_data/Adult-Sports.txt
T1B6A23A77F38617A186C60116725A96DA9BBEC97C73114910E84C817EF30BA64433FBFA        example_data/AfghanistanLiteracyDiscussionPaper.txt
T11971AD5BD348133621131373654F39EBEB18933C63274561487ED169335ACAD17B7AD8        example_data/BBVDlivepr.txt
T117519C67C354133467220270AD2F3CEDF714917DA3568B20582AF42D372681B43375E9        example_data/BBVDpr.txt
T129514417E3C8337006414292395E3AA5F627C4782368B938DD69C26FA325FA50376ADB        example_data/BENEFITS_OF_EXERCISE_for_the_manual.txt
T1A8124B13A70917721ED31250734F17FAABBA876C32144A55982CC12C1646DB8D7BFBFA        example_data/Call-for-abstracts.txt
T13BC1895B330F037657B603A1424E43E173595E68733A9A54486B612B3387C5CE7BA3ED        example_data/ClaremontTrioBio0607.txt
T15C627C17E78923B100070266320F39D3A73EC678B22556147C9D919E7389E7D177BBE6        example_data/CommandGde.txt
T123E118177B1533304A4301C27D5A10E6E77AA578B35A1A01B0FEC11457EACAB9B3BBF9        example_data/DOC-DHHS_Joint_Action_Plan_summary.txt
T189A19B8BE70413734A820238F5AE5697FBA5846673139D94D4AC607D321287983FFFA9        example_data/DanceFestOMAIPressReleaseFeb15_07.txt
T1BDC16B1FC71C22350EB20A7B324C36D8AE7B88690245ED75CD1CD11D3365BEE427E6A5        example_data/ExercisePocketSliderText.txt
T199F1AE07E70927351B2301E2AA0F61E3FB589458736AC9054C5EB12133C3DAD937EEE9        example_data/FF06_press_9-7.txt
T1D7D17E27E748377142220064964DB1F5ABAD963833210DB5349D823E238EDBE837F7D9        example_data/Fitness_Myths.txt
T11061694BF2C4223767037275A14B22CCAB7C5934E3245519D4ED51AC9312FA98B3BEE8        example_data/Global_Warming_Resolution_April_2001.txt
T1AD828B23A25013B51AD3017371DD2AD1F37AC1ACA3253C85648DC53C2B57AB9933A7EB        example_data/GovernorsTaskForceReport.txt
T13B718D67960813745AD201D290591CC2932EC22836E1C5A470CF9ACC1B2F8FDD7FBAE1        example_data/Groove_Yard.txt
T1A7B18037B36C23315F0302B53589A1E6FB78854A33BA8E58C41E926533428F447BA3CD        example_data/HSCPR10306.txt
T1C7D17B3FEB482735479100A57B4D3ED6E7258A3833367AB2AC5CD15C132B67906721F6        example_data/HandoutSportsInjuryPrevention.txt
T182928E1BFB4423351B1341F2111D79CBBB28436CB3624D34789D9269278296C97FEBEA        example_data/IntlChildrensFestRevisedRelease.txt
T198025897735813360E564221A68D63D2F328C83A73BF8B2550BDA16C3687C6953FB5EC        example_data/LOTRLeadRelease.txt
T111A15A2753E603564767032A699EB2D3B325813C6150C964306C6BAD37829B483BF9EA        example_data/Landau-Pickford-Press-Rel.txt
T168617C07E729133966738191C24D27DBB76E5028A3298E5414D75ADB2391C6C933FFEC        example_data/Lotuspike-VBTD-Jenkins.txt
T115824C137B9833324B638255B10A1FD3A738D19433705CA4B8A9421867EFA7853F67EB        example_data/MEDT6491.txt
T151817C2B571613399B63062B920DA5E17355C129333BCD0194FDE26C39178B9873DBEC        example_data/MRbio.txt
T1F6626653A788337246920399AB1E57EAEB19D0BC23555934D8CC852C350AC6643FFFF5        example_data/Meeting_the_Challenges.txt
T137224A17A7441F7212D32031764B2AEEE729C96CB3915C659889B12C131C775833BBEA        example_data/Memo_Ehrlich_1103.txt
T1DAD18D2BFB2013660E520228F54E22E5EF798A791362A74178D5812C3396CBD27FD5F8        example_data/Midwestern_Rock_Garden.txt
T1A271690B978527B58A6301F9166C25E5D399C92C332989208D19812D330AABD47BBFE6        example_data/Newsletter_20-1-3.txt
T1F2028E67DB81033449E242A7B14F25EBF725C1393211897470BED2E82359D2993FBAE8        example_data/Reviews-GrooveYard.txt
T14FB1663FD308233447220276B64E61E5BB40D32DD3605572689E94BD33DAD7AD2772E8        example_data/SAR_Dog_Avalanche_promo.txt
T10CB27F27B3D0272653E3016AD81516C3DA3B9149B3040975B47EC278278EB5B63BA7BF        example_data/SCE_3320.txt
T1A161686FF77463B91F828264B19C18E09F6D85247BF680E5B02E536D038585A07FA7B8        example_data/Shopping.txt
T1B65144DF581881590D92CEB572FEBBC5AA3201E0C0C586D4FC9D8CB12715A23B914FE1        example_data/SoundResources.txt
T1F3C1272B67C823B156430214A51A95DAEF38E9283320A539FCDE446C3B1D6561B3BBB6        example_data/Summer_Math_Syllabus.txt
T1DF125A7B778413B106B34504992D66EAE728C17923665C21F4CC826C370A9B647FEBF9        example_data/TeachingMathCameroon.txt
T11BC105871A8433368ED351FCA23A18ACF3B1C5B9372A90554C4E4359B6999290B3F6F2        example_data/The-Global-Warming-Response-Act.txt
T16DC17D87BB4413790F1102E1AA9E61E9F7500A75B3B24F24C4AED0642742C7E2BF97EC        example_data/WPTC2006Season.txt
T1EA427C1FE64C233A09103236364F77DEAB27886CD3141A65AC1D925D3769F9E037A8E9        example_data/Week3.txt
T1D2525A27E74823210B530216761F76E6B73BC42D6315AE20E49DD26C2359E6807FBBDE        example_data/aReportOfTheSurgeonGeneral.txt
T1BE225A22F35D13A58B518369911762D3BB71C249F3386A84C49FA2253747CE8827B7FA        example_data/ashaam2.txt
T1F37179A7BB911376899301B1715E31F2FB68D17CA39991428C2DD1AC32638B9127FECD        example_data/billboard-021205.txt
T1F2715B67761C0379579203A3519F22F5AB20822C73B68E80405D64A57383A7CA6FEAFD        example_data/bio.txt
T1F5222E336FC023568A838147758957EEF22DD9A833145645DC0783A9260F6F9A3BFEC6        example_data/cme2i05.txt
T1FA12A03BE7450375C66B8281224E55D6B639815CF3768A111CAE9038378B467C77BBFC        example_data/didges_brew_band_info.txt
T13A726A1BB35473B91BA38365225DAAF1D7364068B33A0964F46EC1202791C7693FE7EC        example_data/exploring_dallas.txt
T1D3C13A63B355173A82930175626B36E1DBFF8474B3948905684DA2AC834AC68D33F7EE        example_data/exxonmobil_climate_change_backgrounder.txt
T1CB426A53BB0C3337426362B59A5EB6D0F32A842CF3513A70845DD1A92385EB447B76DA        example_data/fitnessTerms.txt
T1E1126C1FE35033A00B2F4166B74626DAEB26022C73970969742D823C73A56AF533B7F5        example_data/gogirls.2003-10.txt
T159427D1FA381233B496102B5770EA2CBEB15C0BC5369A571A45EC12E3367D7C937B9E8        example_data/greenhouse.txt
T102925B1BF74413364D6731B1265FA5DFE72EC628F3650A71E8AC802C2258EA493363DD        example_data/lac_rfp_inner-cities.txt
T1DEC16E0BB344033B4EE352D0318E05AEF7A9907957F2CA94D46CC0593A93C78617ABED        example_data/ledger091505.txt
T161120297328C273521B3025B764FA4D6BB318465D30E4A0028ED99FC936BFA8D7372D9        example_data/modelgeneralnotice.txt
T16B827C5FB74C13B229610595332FAAFAF723C1BC1316916C9C1D802C1365F6A927F6DA        example_data/parentarticle.txt
T10171683E9341233EAB334611369C1FE3761C913AE322885574DD35282F5A62F9ABD3D9        example_data/record_voting.txt
T1E2F16E0B638A3335682A4157B35F11EAB366A1247265C1E0905D333F23DDEEA813B7D7        example_data/rkc_flyer.txt
T18541144FF38513711827423599BD78D37A78D378B304494550B884EA02A392A47FFBDC        example_data/sisters_pressrelease.txt
T1F8A0220C0F8C0023CB880800CA33E88B8F0C022AB302C2008A030300300E8A00C83AAC        example_data/small.txt
T1C6A022A2E0008CC320C083A3E20AA888022A00000A0AB0088828022A0008A00022F22A        example_data/small2.txt
T1A592285B7B5952365703327AB1471ACE6625002DF212CC39A0A9AC383781D6DAFF278E        example_data/socy642_f02_milkie.txt
T1E481582B33C423B545639375535F5AFBB74EC694421183F0A89EC43E735698C11B9AE8        example_data/spanish_place_namesA.txt
T15C127E29F30903720143429821CF67F2B75691E8D2720365B86CCA3EF693EE951F4CEA        example_data/spanish_place_namesB.txt
T11C528E27BB8813744A5303ABB60D39C1F76981BC7650A6A164DE925DB30EC740377BEA        example_data/sports_massage_and_recovery.txt
T16A827C73A39423714BA30263A56E43D7F77F846C22698960A44DC13D624ADF4437FBEA        example_data/srilanka.txt
T13602585BF340635353631252A22F8CDACB14F13DB3927931286CD13D2679E26477BBA6        example_data/survey_mathtch.txt
T198A27C7FA280037117E30126755D9BCAF72A906C337548B9F85E507C331BA28677BBA6        example_data/syllabus.txt
T15C329E6FF70463252A410264660E22DFE3A54038F3AD8B605C3E901403D6F9B5ABB7FE        example_data/victim.txt
T157026A5FF780237092AA01D5670E21E2E36A942CB3265960945D922C3366FFD837B7FD        example_data/website_course_descriptors06-07.txt

Tested versions 4.8.2, and 4.11.2, and confirmed the issue to appear on both versions on big-endian ppc64, hppa, and sparc, with identical results.

trend-mark-chang commented 1 year ago

Hi @matoro

tlsh supports big-endian architectures but you will need to add a compiler option in tlsh/CMakeLists.txt

add_compile_options("-D__SPARC")

The definition is just changing structure order, can be used to all big-endian architectures

matoro commented 1 year ago

Hi @matoro

tlsh supports big-endian architectures but you will need to add a compiler option in tlsh/CMakeLists.txt
add_compile_options("-D__SPARC")
The definition is just changing structure order, can be used to all big-endian architectures

Hi, I just tested and that indeed does the trick - thank you so much for the hint!

trendmicro / tlsh

Unit tests fail on big-endian architectures #131