Arp-G / csv2sql

A blazing fast fully-automated CSV to database importer
MIT License
54 stars 4 forks source link

MySQL error: row size to large #25

Open sagnick-solo opened 2 years ago

sagnick-solo commented 2 years ago

`AN ERROR OCCURED AND FURTHER PROCESSING WAS STOPPED:

%MyXQL.Error{connection_id: 175, message: "(1118) (ER_TOO_BIG_ROWSIZE) Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs", mysql: %{code: 1118, name: :ER_TOO_BIG_ROWSIZE}, statement: "CREATE TABLE accur_recruiting.Contact (Id VARCHAR(100), IsDeleted BIT, MasterRecordId VARCHAR(100), AccountId VARCHAR(100), Salutation VARCHAR(100), FirstName VARCHAR(100), LastName VARCHAR(100), RecordTypeId VARCHAR(100), OtherStreet VARCHAR(100), OtherCity VARCHAR(100), OtherState VARCHAR(100), OtherPostalCode VARCHAR(100), OtherCountry VARCHAR(100), OtherLatitude VARCHAR(100), OtherLongitude VARCHAR(100), OtherGeocodeAccuracy VARCHAR(100), MailingStreet VARCHAR(100), MailingCity VARCHAR(100), MailingState VARCHAR(100), MailingPostalCode VARCHAR(100), MailingCountry VARCHAR(100), MailingLatitude VARCHAR(100), MailingLongitude VARCHAR(100), MailingGeocodeAccuracy VARCHAR(100), Phone VARCHAR(100), Fax VARCHAR(100), MobilePhone VARCHAR(100), HomePhone VARCHAR(100), OtherPhone VARCHAR(100), AssistantPhone VARCHAR(100), ReportsToId VARCHAR(100), Email VARCHAR(100), Title TEXT, Department VARCHAR(100), AssistantName VARCHAR(100), LeadSource VARCHAR(100), Birthdate VARCHAR(100), Description TEXT, OwnerId VARCHAR(100), HasOptedOutOfEmail BIT, HasOptedOutOfFax BIT, DoNotCall BIT, CreatedDate VARCHAR(100), CreatedById VARCHAR(100), LastModifiedDate VARCHAR(100), LastModifiedById VARCHAR(100), SystemModstamp VARCHAR(100), LastActivityDate VARCHAR(100), LastCURequestDate VARCHAR(100), LastCUUpdateDate VARCHAR(100), EmailBouncedReason TEXT, EmailBouncedDate VARCHAR(100), Jigsaw VARCHAR(100), JigsawContactId VARCHAR(100), IndividualId VARCHAR(100), ts2__Picture_Id__c VARCHAR(100), Legacy_ID__c VARCHAR(100), Legacy_File_ID__c VARCHAR(100), Legacy_File_Resume_ID__c VARCHAR(100), ts2__Hide_Photo__c BIT, ts2__Legacy_ContactID__c VARCHAR(100), ts2__Legacy_DocumentID__c VARCHAR(100), ts2__ReferrerEmailMessage__c VARCHAR(100), ts2__ReferrerEmailSubject__c VARCHAR(100), ts2__SessionKey__c VARCHAR(100), ts2__Education_School_1__c VARCHAR(100), ts2__Education_School_2__c VARCHAR(100), ts2__JobNotificationsSummary__c BIT, ts2__EEO_Disabled__c VARCHAR(100), ts2__EEO_Gender__c VARCHAR(100), ts2__EEO_Race__c VARCHAR(100), ts2__EEO_Veteran_Status__c VARCHAR(100), ts2__Referral_Lookup__c VARCHAR(100), ts2__Verified_Key__c VARCHAR(100), ts2__Verified__c BIT, ts2__Geo_Location_Passed__c BIT, ts2__Latitude__c DOUBLE, ts2__Longitude__c DOUBLE, ts2__Resume_Last_Updated__c VARCHAR(100), ts2__Verified_Date__c VARCHAR(100), test__c VARCHAR(100), accur_candidates_industries__c VARCHAR(100), accur_candidates_functions__c VARCHAR(100), accur_candidates_linkedin_viadeo__c TEXT, accur_candidates_facebook__c VARCHAR(100), accur_candidates_web__c TEXT, accur_candidates_Travel_Retail_Duty_Free__c BIT, accur_candidates_Salary_Range__c VARCHAR(100), accur_candidates_language_list__c VARCHAR(100), accur_candidates_Excel_Level__c VARCHAR(100), accur_candidates_Traveling_ability__c VARCHAR(100), accur_candidates_US_Work_Authorization__c VARCHAR(100), accur_candidates_Video_Resume__c VARCHAR(100), accur_currency__c VARCHAR(100), accur_CV__c INT, accur_Industry_1__c VARCHAR(100), accur_Industry_2__c VARCHAR(100), accur_Industry_3__c VARCHAR(100), accur_Function_1__c VARCHAR(100), accur_Function_2__c VARCHAR(100), accur_Function_3__c VARCHAR(100), Salary_Range_2__c VARCHAR(100), accur_Recommended_by__c TEXT, accur_Relocation__c TEXT, accur_Email_Pro__c VARCHAR(100), Percentile_Rankings__c VARCHAR(100), Skype_username__c VARCHAR(100), Contact__c VARCHAR(100), Candidate_Contact_Profile__c VARCHAR(100), Excel_Test_Result__c VARCHAR(100), General_Comments__c VARCHAR(100), Scenario_Date__c VARCHAR(100), TheDecisionMakers_co_uk__c VARCHAR(100), Excel_Test_Date__c VARCHAR(100), Direct_Office_Line__c VARCHAR(100), Office_Mobile__c VARCHAR(100), Company_Extension__c VARCHAR(100), Internal_ITW__c BIT, ts2__Facebook_Profile__c VARCHAR(100), ts2__LUID__c VARCHAR(100), ts2__LinkedIn_Profile__c TEXT, ts2__MailingCountryText__c VARCHAR(100), ts2_" <> ...}

Arpan-Kreeti commented 2 years ago

Note to self:

We must dynamically infer the row size by finding the max data size in that column. So the varchar_limit the user specifies must only be used to decide when to switch from varchar to text.

Arpan-Kreeti commented 2 years ago

Also this can be an enhancement but if row size exceeds max MySQL row size length then maybe changes few varchars to text?

Arpan-Kreeti commented 2 years ago

The empty columns should be like varchar(1) to avoid this