Unique violations during XML import

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Set up a clean copy of the latest CandyDollDB from SVN.
2. Log in and click 'Process XML'
3. Watch as a 10+ screens long error message fills your screen

What is the expected output? What do you see instead?
Expected would be no error message at all.

Peculiar detail during my tests is that the errors occur from Adriana E to 
Sonya M, but not further.

Original issue reported on code.google.com by fwp...@gmail.com on 21 Feb 2012 at 11:50

[GoogleCodeExporter]

On a different server, I cannot reproduce this error.

Perhaps there was still an old database on the server? But then why was it not 
dropped by the setup-script?

I've marked this issue as 'Started' and 'medium'. Any further tests would be 
appreciated.

FWieP

Original comment by fwp...@gmail.com on 21 Feb 2012 at 5:49

• Changed state: Started
• Added labels: Priority-Medium
• Removed labels: Priority-High

[GoogleCodeExporter]

I tried it and was not able to replicate the problem. 

I am leaning towards an old database left behind. r292 allowed for custom DB 
names, so possibly there was a reference to the old DB left behind

Original comment by mranimos...@gmail.com on 22 Feb 2012 at 4:56

[GoogleCodeExporter]

dont know if it helps, but I am running Apache 2.2.17 with PHP 5.3.5 and MySql 
5.5.8, installed through XAMPP 1.7.4 on Windows 7

Original comment by mranimos...@gmail.com on 22 Feb 2012 at 4:59

[GoogleCodeExporter]

I'm at work now, and have reproduced the problem. It's a virtual-pc running 
Ubuntu Lucid, Apache 2.2.14 and MySQL 5.1.41.

The database is being dropped and recreated nicely.

When I start the Process-XML on the command line, the progress indicator rises 
above 100%, to about 170%.

After that, I inspect the inserted records (models) with PhpMyAdmin. I see that 
up until Sonya M (id = 77) all is well, but then the AUTO-INCREMENT jumps 
forward. This indicates the UNIQUE errors during import.

I validate and use the latest XML-file, use the latest setup_data.php from SVN 
and have verified that the database is clean before importing.

I suspect setup_data.php is looping more than it should.

FWieP

Original comment by fwp...@gmail.com on 22 Feb 2012 at 11:50

[GoogleCodeExporter]

this was added to setup_data recently, cannot find the revision number, but i 
know it was recent

        // Provide a one-model-only import for impatient developers
        if($Model2Process->getID() && $ModelID && $Model2Process->getID() !== $ModelID)
        { continue; }
try removing it and then see if the problem is re-creatable 

Original comment by mranimos...@gmail.com on 22 Feb 2012 at 10:06

[GoogleCodeExporter]

Tried your suggestion, to no avail.

Tried completely removing the cd-folder and doing a new check-out of r328. 
Setup, import, BOOM: unique violations after ID=77.

Curious detail: Today a new model (Abrora H) was added to the XML, thus 
shifting Sonya M up by 1, now the import still BOOMs at ID=77, being Sofiya V...

This is getting weirder by the minute... I'll give it some more tries, but I'm 
beginning to suspect a messed-up PHP-MySQL-combination...

FWieP

Original comment by fwp...@gmail.com on 23 Feb 2012 at 11:30

[GoogleCodeExporter]

Looked for differences between this server and my dev-machine. Could (one of) 
these be causing problems?

- Core PHP: 5.3.2-1ubuntu4.14
      mine: 5.3.6-13ubuntu3.6

- LibXML: 2.7.6
    mine: 2.7.8

- Mysql client: 5.1.41
          mine: 5.1.58

- Mysqli client: 5.1.41
           mine: 5.1.58

- SimpleXML: Revision: 293036
       mine: Revision: 308262

FWieP

Original comment by fwp...@gmail.com on 23 Feb 2012 at 11:45

[GoogleCodeExporter]

will try a new checkout and install again after work

Original comment by mranimos...@gmail.com on 23 Feb 2012 at 11:59

[GoogleCodeExporter]

checked out r329, processed XML and no errors showed. Dropped the database and 
setup again, this time processing the XML on the CLI, went to 100% with no 
errors. As you can see below, most of my versions are below yours, so I don't 
know if that would be the problem since it obviously worked fine for awhile I 
assume

- Server: Apache/2.2.17 (Win32) on Windows 7 (installed via XAMPP 1.7.4)

- Core PHP: 5.3.5
     yours: 5.3.6-13ubuntu3.6

- LibXML: 2.7.7 
   yours: 2.7.8

- Mysql client: 5.0.7-dev - 091210 - Revision: 304625
         yours: 5.1.58

- Mysqli client: 5.0.7-dev - 091210 - Revision: 304625
          yours: 5.1.58

- SimpleXML: Revision: 302715
      yours: Revision: 308262

Original comment by mranimos...@gmail.com on 24 Feb 2012 at 9:12

[GoogleCodeExporter]

More testing on this very weird virtual-pc... When I comment out the last 25% 
of the XML, so that only ~75 models will be imported, that's exactly what 
happens. Without errors.

When I put ONE more model in the XML, the CLI-progress indicator loops from 
0-200%. As expected, the last AUTO-INCREMENT has junped forward.

First I suspected a problem in the InnoDB handling of NULLable columns in 
UNIQUE-constraints. So, I modified the Model-class to give Lena and Oksana a 
dummy lastname. No go...

Remark: There are the correct number of 99 models present in the database. So 
there definitely is something to these NULLable lastnames...

Original comment by fwp...@gmail.com on 24 Feb 2012 at 11:40

[GoogleCodeExporter]

speaking of Lena, who is she, i have yet to see anything on her, same goes for 
Alina N

Original comment by mranimos...@gmail.com on 24 Feb 2012 at 11:53

[GoogleCodeExporter]

Now with provable numbers. I ran three import runs, each starting with a clean 
DB.

1) Commented out the last models, so that from AbroraH -> SilviyaR would be 
imported. This yielded 100% and 76 models. All is well.

2) De-commented one more model. The range became AbroraH -> SofiyaV. This 
yielded 200% and 79 records. Lena and Oksana were added twice! Both these sets 
had a jumped model_id.

3) Ran the same run (AbroraH -> SofiyaV) with dummy lastnames for Lena and 
Oksana. This gave me 77 records but still 200%. One abbreviation that would fit 
my reaction would be w.t.f..?

I have no idea what is going wrong here, but I at least have some data to 
research, look up, ponder about.

FWieP

Original comment by fwp...@gmail.com on 24 Feb 2012 at 12:08

[GoogleCodeExporter]

Alina N appeared some months ago on the CD-site, with one sample image. Never 
seen a set of her.

Lena is the fourth girl in the SP_4Dress special set. When I learned her name, 
I thought it right to add her to the collection. No sets of her, either.

FWieP

Original comment by fwp...@gmail.com on 24 Feb 2012 at 12:10

[GoogleCodeExporter]

More testing on that less than fortunate virtual-pc... Finally some progress!

Ran a new run with all source code updated to v1.6, including setup_data.php 
and its XML-file. Now it runs to about 140%, then finishes. Then it spits out 
the following error:

 zend_mm_heap corrupted

Google searches revealed some indication to garbagecollection, but I haven't 
been able to find the bug as such... At least we now know there IS a bug to be 
found.

FWieP

Original comment by fwp...@gmail.com on 27 Feb 2012 at 12:02

[GoogleCodeExporter]

Removed and purged PHP, re installed PHP, no go.

Removed and purged MySQL, re installed MySQL, no go.

The zend_mm_heap error never showed up again...

Original comment by fwp...@gmail.com on 2 Mar 2012 at 11:39

[GoogleCodeExporter]

Ran another couple of runs: Remove config, re-setup (being sure that the DB was 
being dropped in between), import using CLI of GUI.

First run with most up to date XML. It processed, then complains about UNIQ_SET 
which is being violated for model_id=77. Later I see the following range of 
model_ids:

AbroraH       SofiyaV  Lena   Oksana  SonyaM        ZinaB VIP Promo Interv.
   1    .....   77      120     139     154  ......  173  174  175   176

During this same run on the command line, I got a 'segmentation fault'.

When I comment all but the first 76 models, all is well and no errors are 
thrown.

I think that Lena and Oksana are imported twice because of some NULL-check 
going wrong. I'd like to get this #$&^!-76 issue fixed first.

Original comment by fwp...@gmail.com on 6 Mar 2012 at 12:13

[GoogleCodeExporter]

try adding a fake last name for Lena and Oksana, see if that works, if so then 
it definitely is a bad Null-check somewhere

Original comment by mranimos...@gmail.com on 7 Mar 2012 at 6:49

[GoogleCodeExporter]

Tried your suggestion, added 'DUMMY' as last name for Lena and Oksana.

AbroraH       SofiyaV   SonyaM        ZinaB VIP Promo Interv.
   1    .....   77        154  ......  173  174  175   176

This is getting weirder and weirder. I'm open to any other suggestions?

Original comment by fwp...@gmail.com on 8 Mar 2012 at 10:20

[GoogleCodeExporter]

This is the list of errors, as thrown when one clicks on 'Process XML'.

Original comment by fwp...@gmail.com on 8 Mar 2012 at 11:39

Attachments:

• issue_10_unique_violations.txt

[GoogleCodeExporter]

Dropped and set up once again, now without the NULLable column 'set_prefix' in 
UNIQ_SET. Same result, same errors.

Original comment by fwp...@gmail.com on 8 Mar 2012 at 12:06

[GoogleCodeExporter]

One more run, this time with SharlottaS sets 2-33 removed: Progress!

AbroraH       SonyaM  Lena   Oksana  SundiZ        ZinaB VIP Promo Interv.
   1    .....   78     121     140     156  ......  174  175  176   177

It seems that the total amount of sets plays a part in this weird error. This 
weekend I'll investigate some more, at home.

Original comment by fwp...@gmail.com on 9 Mar 2012 at 12:26

Attachments:

• issue_10_unique_violations_small_SharlottaS.txt

[GoogleCodeExporter]

don't know if you tried this yet, but try commenting out the VIP, Promotion & 
Interview parts of the XML, or adding a blank last name attritube, then 
re-running it becuase I noticed there is no last name attribute for those in 
the XML

Original comment by mranimos...@gmail.com on 10 Mar 2012 at 2:24

[GoogleCodeExporter]

Have been debugging this issue now for 5 hours non-stop. Filled in 
dummy-lastnames for Lena, Oksana and the 3 specials. The foreach-loop of all 
SimpleXML Models begins anew after having processed its 78th node. In the 
second run it processes model 1-101, as it should. The importing code normally 
requests all Models, Sets and Dates ONCE at the start of the run. 

During the second run, the first 78 models, their sets and their dates produce 
the loads and loads of UNIQUE-violations. A temporary work-around would be to 
add the freshly inserted Models, Sets and Dates to these arrays ($ModelsInDb, 
$SetsInDb and $DatesInDb). But this not solving the problem, just hiding it. 
And, this run takes ages when the first 78 models are processed a second time...

I'm going to simplify/strip the XML to a 'simple' test case that can be easily 
brought to the attention of more knowledgeable people, perhaps a problem in 
SimpleXML ? I have no idea...

Original comment by fwp...@gmail.com on 10 Mar 2012 at 8:06

[GoogleCodeExporter]

All right, one more hour of debugging. I'm beginning to suspect some magic 
limit of SimpleXML, PHP or a combination of both.

Models are inserted nicely, they get model_id's 1-13. Sets however:

Set01    Set1239  Set1240     Set1300
 1 .....  1239     1278 ....  1338

This time the same thing happens as earlier from SofiyaV onwards, but around 
the 1239th set. I still can't explain this. Will try a direct approach of 1 
model with 1239 sets. See if that makes it go woozy...

Original comment by fwp...@gmail.com on 10 Mar 2012 at 9:16

Attachments:

• issue_10_13-models-with-100-sets-each.xml
• issue_10_run-with-13-models-and-100-sets-each.txt

[GoogleCodeExporter]

1239 seems to be the magic number of this error. All sets have set_ids 
corresponding to their names, but an enormous load of errors is thrown because 
of a double run of the XML.

Set1     Set1239
  1 ..... 1239

I did a run with 1238 sets (removed last set-node from XML), and all went well, 
without any errors.

Enough for today, I'm going to bed.

Original comment by fwp...@gmail.com on 10 Mar 2012 at 9:34

Attachments:

• issue_10_1-model-with-1239-sets.xml
• issue_10_run-with-1-model-and-1239-sets.txt

[GoogleCodeExporter]

Once more a few hours of debugging at home, and just now a re-run on that 
particular virtual pc. Progress? Yes. Solution? No.

At home I determined that after the processing of model #78, the XML loop 
starts over again and causes loads of DUPLICATE KEY errors. When I commented 
out most of setup_data.php, it ran through flawlessly. So I de-commented one 
piece of code at a time and discovered the evil line (around line 124):

 $CacheImages = CacheImage::GetCacheImages(sprintf('index_id = %1$d', $Model2Process->getID()));

I changed it into 

 $CacheImages = array();

This did the trick! Why? I don't know. I can only summarize that if this code 
runs, this machine goes frenzy after 78 models/12xx sets/13xx XML nodes. I'm 
open to suggestions, and most of all a logical explanation.

Anyone, please?

Original comment by fwp...@gmail.com on 12 Mar 2012 at 11:49

[GoogleCodeExporter]

Traced it down some more, but not quite nailed it - yet.

After 78 models, the GetCacheImages causes the XML-loop to repeat itself. I 
have tracked it into the $db->Select() function. There something goes wrong in 
the filling of the FieldInfo array. It is later used to cast the db-values to 
their corresponding datatypes (currently only int and string).

If I comment out the call to mysql_fetch_field() and set the $db-FieldInfo to 
NULL, all is well and no errors are thrown. This looks promising, I'll put more 
time in it, when I'm able to.

Original comment by fwp...@gmail.com on 13 Mar 2012 at 12:52

[GoogleCodeExporter]

Finally found it, and fixed it in r381.

I did not reset the $db->FieldInfo array when making a new SQL-query. This 
array got bigger, and bigger, and bigger... Up to about 1024, where I think it 
made PHP kick the bucket and go crazy.

When I debugged it using a conditional breakpoint, after 1230 calls to the 
$db-Select,  the array's count was about 1000. When I noticed this, I added a 
reset statement - as I seem to have done from the beginning for the $db's 
result-property.

Well, this is one for (my personal) recordbooks. I'm glad it's finally fixed 
:-) Thanks for thinking along!

Original comment by fwp...@gmail.com on 13 Mar 2012 at 7:14

• Changed state: Fixed