Closed mdtro closed 7 years ago
I have never seen a BOOM before.
It's possible memory is not being freed in a timely manner, but I've not noticed this behavior before. A vast majority of the allocation should be done prior to the first record being generated. I am surprised you are getting this far off it is a memory issue.
I will try to run some tests tomorrow to see if I can identify a leak.
I've not run this on a machine with less than 6GB RAM, so it is possible I've missed something that is triggered on your 2 GB VM. I have generated tens of millions of records from a single process without problems, though.
The problem is fixed. It turns out to be the way the trade dictionary was (not) being copied before elements were being removed from it.
I just generated 250k records without hitting the " BOOM ".
On Mon, Jun 19, 2017 at 4:56 PM, mdtro notifications@github.com wrote:
Hi,
When I pass the argument to datagen.py it fails to generate the specified amount. In the generated records, I get a " BOOM ".
My command: python3 datagen.py 1000 > ~/records.json
Output:
{"name": {"first": "JUNE", "last": "CHOU", "middle": "S", "full": "CHOU JUNE S"}, "addresses": [{"postalcode": "48371\n", "street1": "765 S LAPEER RD.", "state": "MI", "phones": ["3440771404", "0697395443"], "city": "OXFORD", "street2": "", "street3": ""}, {"postalcode": "68137\n", "street1": "4444 S 108TH ST", "state": "NE", "phones": ["5543653927", "3966162771"], "city": "OMAHA", "street2": "", "street3": ""}, {"postalcode": "44113\n", "street1": "1455 W 29TH ST", "state": "OH", "phones": ["9941571833", "0548851931"], "city": "CLEVELAND", "street2": "", "street3": ""}, {"postalcode": "93901\n", "street1": "295 SOUTH MAIN STREET.", "state": "CA", "phones": ["8766878475", "8629149201"], "city": "SALINAS", "street2": "", "street3": ""}], "trades": [{"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "8779847465", "bank_name": "BANCORP", "account_no": "4565348129949658"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "VISA", "country": "US", "bank_phone": "8006424720", "bank_name": "WELLS FARGO", "account_no": "4312432347094846"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "VISA", "country": "US", "bank_phone": "2035154654", "bank_name": "BANK OF AMERICA", "account_no": "4024005860166661"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "8005243880", "bank_name": "CHASE", "account_no": "5417117584257905"}], "ssn": "574607131", "dob": "19811225", "gender": "F"} {"name": {"first": "VERONICA", "last": "SEAY", "middle": "N", "full": "SEAY VERONICA N"}, "addresses": [{"postalcode": "36752\n", "street1": "1112 ST CLAIR PLACE", "state": "AL", "phones": ["6987332785", "2076364739"], "city": "LOWNDESBORO", "street2": "", "street3": ""}, {"postalcode": "39501\n", "street1": "4420 15TH ST", "state": "MS", "phones": ["9327175078", "6811650569"], "city": "GULFPORT", "street2": "", "street3": ""}], "trades": [{"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "3155399261", "bank_name": "COMMUNITY BANK N.A.", "account_no": "4435890318923207"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "8776182648", "bank_name": "ARVEST", "account_no": "4223298360695724"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "8008624066", "bank_name": "", "account_no": "4401684455465267"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "8772872654", "bank_name": "PNC BANK N.A.", "account_no": "4034973559675504"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "(513)579-5353", "bank_name": "FIFTH THIRD BANK, THE", "account_no": "5408803745729524"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "800-426-1917", "bank_name": "SOUTHLAND CIVIC FEDERAL CREDIT UNION", "account_no": "4730643714699187"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "8006228731", "bank_name": "BANK OF AMERICA", "account_no": "5312508199821923"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "", "bank_name": "VIRGINIACREDITUNION", "account_no": "5344665656405644"}], "ssn": "214144559", "dob": "20141021", "gender": "F"} {"name": {"first": "LINO", "last": "BRINER", "middle": "STEVEN", "full": "BRINER LINO STEVEN"}, "addresses": [{"postalcode": "73105\n", "street1": "200 E HILL ST", "state": "OK", "phones": ["0449866500", "5270897943"], "city": "OKLAHOMA CITY", "street2": "", "street3": ""}, {"postalcode": "44105\n", "street1": "3662 E 65TH STREET", "state": "OH", "phones": ["5869177756", "7320417168"], "city": "CLEVELAND", "street2": "", "street3": ""}], "trades": [{"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "8005318722", "bank_name": "USAA", "account_no": "5438050222773748"}], "ssn": "215589639", "dob": "19400425", "gender": "M"} {"name": {"first": "MARGARITA", "last": "SUNGA", "full": "SUNGA MARGARITA"}, "addresses": [{"postalcode": "92128\n", "street1": "17830 VIA ALTA MIRASOL", "state": "CA", "phones": ["4286465803", "4982938424"], "city": "SAN DIEGO", "street2": "", "street3": ""}], "trades": [{"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "AMEX", "country": "US", "bank_phone": "8002570770", "bank_name": "AMERICAN EXPRESS", "account_no": "372533080763729"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "6069283532", "bank_name": "KENTUCKY FARMERS", "account_no": "5325260989098990"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "6663", "bank_name": "PIONEER VALLEY FEDERAL CREDIT UNION", "account_no": "5150503689516639"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "6093354122", "bank_name": "BELCO", "account_no": "4108433125219776"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "8777771617", "bank_name": "WELLS FARGO", "account_no": "5115592186606340"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "5019822328", "bank_name": "ARKANSAS FEDERAL CREDIT UNION", "account_no": "4402033858341507"}], "ssn": "065642346", "dob": "19580223", "gender": "F"} {"name": {"first": "ZONA", "last": "DOMINGUEZ", "middle": "T", "full": "DOMINGUEZ ZONA T"}, "addresses": [{"postalcode": "17872\n", "street1": "258 OVERLOOK BLVD", "state": "PA", "phones": ["6170042590", "7413606577"], "city": "SHAMOKIN", "street2": "", "street3": ""}], "trades": [{"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "AMEX", "country": "US", "bank_phone": "8882461076", "bank_name": "AMERICAN EXPRESS", "account_no": "377257835981460"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "8667896041", "bank_name": "GE CAPITAL RETAIL", "account_no": "5148656782106049"}], "ssn": "468036772", "dob": "19971126", "gender": "F"} {"name": {"first": "MARY", "last": "ALAMO", "full": "ALAMO MARY"}, "addresses": [{"postalcode": "63368\n", "street1": "7092 S OUTER ROAD 364 APT 278", "state": "MO", "phones": ["0716469422", "1915339829"], "city": "O FALLON", "street2": "", "street3": ""}, {"postalcode": "60611\n", "street1": "500 N MICHIGAN AVE STE 800", "state": "IL", "phones": ["4646272983", "9943909224"], "city": "CHICAGO", "street2": "", "street3": ""}], "trades": [{"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "8003465312", "bank_name": "BANK OF TEXAS", "account_no": "4688813050074997"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "8002427338", "bank_name": "CHASE", "account_no": "4427322141225381"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "VISA", "country": "US", "bank_phone": "8773736192", "bank_name": "USBANK", "account_no": "4186216748846777"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "(504) 278-1650", "bank_name": "REGIONS BANK", "account_no": "4294754344285021"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "VISA", "country": "US", "bank_phone": "877-825-3242", "bank_name": "CREDIT ONE", "account_no": "4447961711204543"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "8008515152", "bank_name": "GLENS FALLS NATIONAL BANK & TRUST COMPANY", "account_no": "4236025904025247"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "8504799601", "bank_name": "GULF WINDS FEDERAL CREDIT UNION", "account_no": "4833328152632807"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "8003428298", "bank_name": "FIRST FINANCIAL CREDIT UNION", "account_no": "5517016432383188"}], "ssn": "054359234", "dob": "19840922", "gender": "F"} {"name": {"first": "M", "last": "MANCIA", "full": "MANCIA M"}, "addresses": [{"postalcode": "15201\n", "street1": "4729 HATFIELD STREET", "state": "PA", "phones": ["5419829147", "7143634957"], "city": "PITTSBURGH", "street2": "", "street3": ""}], "trades": [{"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "8004802265", "bank_name": "HUNTINGTON", "account_no": "5175455160046542"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "AMEX", "country": "US", "bank_phone": "8005284800", "bank_name": "AMERICAN EXPRESS", "account_no": "373190533822797"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "713-627-4000", "bank_name": "MEMBERSOURSE CREDIT UNION", "account_no": "4806916100057103"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "VISA", "country": "US", "bank_phone": "8666339958", "bank_name": "CHEVRON FEDERAL CREDIT UNION", "account_no": "4463321295442260"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "AMEX", "country": "US", "bank_phone": "8009260143", "bank_name": "AMERICAN EXPRESS", "account_no": "372321092532114"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "9208307200", "bank_name": "FIRST COMMUNITY CREDIT UNION", "account_no": "4423264632467025"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "8009962638", "bank_name": "BANK OF THE WEST", "account_no": "5472162017349131"}], "ssn": "524047634", "dob": "19610627", "gender": "F"} {"name": {"first": "MATTHEW", "last": "GOUKER", "middle": "CHARLES", "full": "GOUKER MATTHEW CHARLES"}, "addresses": [{"postalcode": "65243\n", "street1": "19440 N BARBER RD", "state": "MO", "phones": ["9012002484", "0446476869"], "city": "CLARK", "street2": "", "street3": ""}, {"postalcode": "79845\n", "street1": "205 E EL PINE ST", "state": "TX", "phones": ["0218010421", "7769368059"], "city": "PRESIDIO", "street2": "", "street3": ""}], "trades": [{"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "VISA", "country": "US", "bank_phone": "8004321000", "bank_name": "BANK OF AMERICA", "account_no": "4356873538531143"}, {"brand": "", "prepaid": "", "type": "DEBIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "800-975-4722", "bank_name": "HSBC", "account_no": "5443680899900419"}, {"brand": "", "prepaid": "", "type": "CREDIT", "scheme": "MASTERCARD", "country": "US", "bank_phone": "8668886080", "bank_name": "PAYPAL", "account_no": "5581589449848796"}], "ssn": "353381471", "dob": "19940509", "gender": "M"} BOOM {'brand': '', 'prepaid': '', 'type': 'DEBIT', 'scheme': 'MASTERCARD', 'country': 'US', 'bank_phone': '8003428298', 'bank_name': 'FIRST FINANCIAL CREDIT UNION', 'account_no': '5517016432383188'}
I am running this inside an Ubuntu VM with 2GB of RAM provisioned to it. Any ideas what could be causing this? What can I do to help?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jcwoods/datagen/issues/1, or mute the thread https://github.com/notifications/unsubscribe-auth/AG0hX6oxvs3E5y0UjbFSs9aOmYttvv9Nks5sFuB6gaJpZM4N-x8r .
Ah great! I'll give this another shot with the updated code and let you know.
Hi,
When I pass the argument to datagen.py it fails to generate the specified amount. In the generated records, I get a " BOOM ".
My command:
python3 datagen.py 1000 > ~/records.json
Output:
I am running this inside an Ubuntu VM with 2GB of RAM provisioned to it. Any ideas what could be causing this? What can I do to help?