Unable to add duplicate unit

[JohnnyKing94]

Why can't I add duplicate units?? They have different ids and they're intended to be so in the lang file i have, how can i add these lines without having this warning?

2015-09-23 16:36:39,397 WARNING Unable to add duplicate unit: my-unit-text

Thx in advance

[julen]

Can you maybe share these duplicate units as a start?

[JohnnyKing94]

This is only a part of the log, cause it has so many strings in it

2015-09-23 16:47:17,858 WARNING Unable to add duplicate unit: Morkuldin Protector
2015-09-23 16:47:17,863 WARNING Unable to add duplicate unit: Morkuldin Sentinel
2015-09-23 16:47:17,867 WARNING Unable to add duplicate unit: Alinon the Alchemist^M
2015-09-23 16:47:17,871 WARNING Unable to add duplicate unit: Orsinium Guard^m
[2015-09-23T16:47:17]   system  UA      it      78350   /it/test/it.csv
2015-09-23 16:47:17,887 WARNING Unable to add duplicate unit: Lady Sovelle^F
2015-09-23 16:47:17,893 WARNING Unable to add duplicate unit: Morkul Ravager
2015-09-23 16:47:17,898 WARNING Unable to add duplicate unit: Morkul Ravager
2015-09-23 16:47:17,904 WARNING Unable to add duplicate unit: Lord Ethian^M
2015-09-23 16:47:17,908 WARNING Unable to add duplicate unit: Skordo the Knife^M
[2015-09-23T16:47:17]   system  UA      it      78351   /it/test/it.csv
[2015-09-23T16:47:17]   system  UA      it      78352   /it/test/it.csv
[2015-09-23T16:47:17]   system  UA      it      78353   /it/test/it.csv
[2015-09-23T16:47:17]   system  UA      it      78354   /it/test/it.csv
[2015-09-23T16:47:17]   system  UA      it      78355   /it/test/it.csv
2015-09-23 16:47:17,960 WARNING Unable to add duplicate unit: Alinon the Alchemist^M
[2015-09-23T16:47:17]   system  UA      it      78356   /it/test/it.csv
[2015-09-23T16:47:17]   system  UA      it      78357   /it/test/it.csv
[2015-09-23T16:47:17]   system  UA      it      78358   /it/test/it.csv
[2015-09-23T16:47:17]   system  UA      it      78359   /it/test/it.csv
[2015-09-23T16:47:18]   system  UA      it      78360   /it/test/it.csv
2015-09-23 16:47:18,013 WARNING Unable to add duplicate unit: Winterborn Snow Wizard^m
2015-09-23 16:47:18,018 WARNING Unable to add duplicate unit: Winterborn Snow Wizard^f
2015-09-23 16:47:18,023 WARNING Unable to add duplicate unit: Winterborn Mage^m
2015-09-23 16:47:18,029 WARNING Unable to add duplicate unit: Winterborn Mage^f
2015-09-23 16:47:18,034 WARNING Unable to add duplicate unit: Winterborn Marauder^m
2015-09-23 16:47:18,039 WARNING Unable to add duplicate unit: Winterborn Marauder^f
2015-09-23 16:47:18,044 WARNING Unable to add duplicate unit: Winterborn Archer^m
2015-09-23 16:47:18,050 WARNING Unable to add duplicate unit: Winterborn Archer^f
2015-09-23 16:47:18,054 WARNING Unable to add duplicate unit: Hagraven
2015-09-23 16:47:18,058 WARNING Unable to add duplicate unit: Alinon the Alchemist^M
2015-09-23 16:47:18,060 WARNING Unable to add duplicate unit: Bear
2015-09-23 16:47:18,063 WARNING Unable to add duplicate unit: Bear
[2015-09-23T16:47:18]   system  UA      it      78361   /it/test/it.csv
2015-09-23 16:47:18,077 WARNING Unable to add duplicate unit: Keeper Guardian^m
2015-09-23 16:47:18,082 WARNING Unable to add duplicate unit: Keeper Predator^f
2015-09-23 16:47:18,087 WARNING Unable to add duplicate unit: Eveli Sharp-Arrow^F
[2015-09-23T16:47:18]   system  UA      it      78362   /it/test/it.csv
2015-09-23 16:47:18,103 WARNING Unable to add duplicate unit: Dremora Footsoldier
[2015-09-23T16:47:18]   system  UA      it      78363   /it/test/it.csv
2015-09-23 16:47:18,118 WARNING Unable to add duplicate unit: Dremora Healer
[2015-09-23T16:47:18]   system  UA      it      78364   /it/test/it.csv
2015-09-23 16:47:18,133 WARNING Unable to add duplicate unit: Dremora Assassin
2015-09-23 16:47:18,137 WARNING Unable to add duplicate unit: Dwarven Centurion
2015-09-23 16:47:18,142 WARNING Unable to add duplicate unit: Riekr Warrior
2015-09-23 16:47:18,147 WARNING Unable to add duplicate unit: Riekr Mender
2015-09-23 16:47:18,153 WARNING Unable to add duplicate unit: Riekr Tempest
2015-09-23 16:47:18,158 WARNING Unable to add duplicate unit: Riekr Savage
2015-09-23 16:47:18,159 WARNING Unable to add duplicate unit: Ogre
2015-09-23 16:47:18,160 WARNING Unable to add duplicate unit: Ogre
2015-09-23 16:47:18,161 WARNING Unable to add duplicate unit: Ogre
2015-09-23 16:47:18,162 WARNING Unable to add duplicate unit: Durzog
2015-09-23 16:47:18,162 WARNING Unable to add duplicate unit: Durzog
2015-09-23 16:47:18,167 WARNING Unable to add duplicate unit: Keeper Swordmaster^m
[2015-09-23T16:47:18]   system  UA      it      78365   /it/test/it.csv
2015-09-23 16:47:18,182 WARNING Unable to add duplicate unit: Echatere
2015-09-23 16:47:18,187 WARNING Unable to add duplicate unit: screenshot_characters_website
[2015-09-23T16:47:18]   system  UA      it      78366   /it/test/it.csv
[2015-09-23T16:47:18]   system  UA      it      78367   /it/test/it.csv
2015-09-23 16:47:18,213 WARNING Unable to add duplicate unit: Eveli Sharp-Arrow^F
2015-09-23 16:47:18,216 WARNING Unable to add duplicate unit: Alinon the Alchemist^M
2015-09-23 16:47:18,221 WARNING Unable to add duplicate unit: Corporal Lugrots^M

[JohnnyKing94]

If you wanna know, this is the part of the csv related to the part of the log i posted. As you notice, they have different IDs

"8290981-0-65018","Morkuldin Protector",""
"8290981-0-65019","Morkuldin Sentinel",""
"8290981-0-65020","Alinon the Alchemist^M",""
"8290981-0-65021","Orsinium Guard^m",""
"8290981-0-65022","Morkul Ravager",""
"8290981-0-65023","Lady Sovelle^F",""
"8290981-0-65024","Morkul Ravager",""
"8290981-0-65025","Morkul Ravager",""
"8290981-0-65026","Lord Ethian^M",""
"8290981-0-65027","Skordo the Knife^M",""
"8290981-0-65028","Orzik the Storm-born",""
"8290981-0-65029","Talonclaw",""
"8290981-0-65030","Fleshrender",""
"8290981-0-65031","Mercedene the Cursed^F",""
"8290981-0-65032","Great Zoraluma",""
"8290981-0-65033","Alinon the Alchemist^M",""
"8290981-0-65034","Janavynn",""
"8290981-0-65035","Wuhkusto",""
"8290981-0-65036","Millengor",""
"8290981-0-65037","Winterborn Butcher^m",""
"8290981-0-65038","Winterborn Butcher^f",""
"8290981-0-65039","Winterborn Snow Wizard^m",""
"8290981-0-65040","Winterborn Snow Wizard^f",""
"8290981-0-65041","Winterborn Mage^m",""
"8290981-0-65042","Winterborn Mage^f",""
"8290981-0-65043","Winterborn Marauder^m",""
"8290981-0-65044","Winterborn Marauder^f",""
"8290981-0-65045","Winterborn Archer^m",""
"8290981-0-65046","Winterborn Archer^f",""
"8290981-0-65047","Hagraven",""
"8290981-0-65048","Alinon the Alchemist^M",""
"8290981-0-65049","Bear",""
"8290981-0-65050","Bear",""
"8290981-0-65051","Q5318 - Captured Alinon Theat",""
"8290981-0-65052","Keeper Guardian^m",""
"8290981-0-65053","Keeper Predator^f",""
"8290981-0-65054","Eveli Sharp-Arrow^F",""
"8290981-0-65055","Dremora Footsoldier",""
"8290981-0-65056","Dremora Footsoldier",""
"8290981-0-65057","Dremora Healer",""
"8290981-0-65058","Dremora Healer",""
"8290981-0-65059","Dremora Assassin",""
"8290981-0-65060","Dremora Assassin",""
"8290981-0-65061","Dwarven Centurion",""
"8290981-0-65062","Riekr Warrior",""
"8290981-0-65063","Riekr Mender",""
"8290981-0-65064","Riekr Tempest",""
"8290981-0-65065","Riekr Savage",""
"8290981-0-65066","Ogre",""
"8290981-0-65067","Ogre",""
"8290981-0-65068","Ogre",""
"8290981-0-65069","Durzog",""
"8290981-0-65070","Durzog",""
"8290981-0-65071","Keeper Swordmaster^m",""
"8290981-0-65072","Briarheart Tree Tender^m",""
"8290981-0-65073","Echatere",""
"8290981-0-65075","screenshot_characters_website",""
"8290981-0-65076","MktSrn_Male01",""
"8290981-0-65077","Mkt_Dreugh",""
"8290981-0-65078","Eveli Sharp-Arrow^F",""
"8290981-0-65079","Alinon the Alchemist^M",""
"8290981-0-65080","Corporal Lugrots^M",""

Thank you in advance :)

[dwaynebailey]

This would be a limitation of the CSV file support. Pootle needs a unique getid() from CSV to see this as unique. In this case, even with unique location, the unit is not seen as unique by Pootle as it doesn't have a unique getid(). At least that is how I understand it.

[JohnnyKing94]

So, what should i do? I need to use absolutely .csv for this project, it's more manageable than other formats... at least for this

[unho]

As @dwaynebailey told this seems to be csvunit not returning different results for getid when the units have the same text but a different context. My guess is that csvunit.context is not properly populated when the CSV file is parsed, and thus https://github.com/translate/translate/blob/master/translate/storage/csvl10n.py#L135-L144 doesn't return different values. So the fix should be on Translate Toolkit, since Pootle relies on it for knowing if a unit has duplicates or not: https://github.com/translate/pootle/blob/b0439ae3/pootle/apps/pootle_store/models.py#L1569-L1573

[JohnnyKing94]

Should i post there or is the same team project and you will handle it??

[unho]

Please create a new issue for Translate Toolkit linking to this one. After that this one can be closed.