coderholic / django-cities

Countries and cities of the world for Django projects
MIT License
921 stars 374 forks source link

Is there anyway to only import citynames/postalcodes for united states? #74

Closed donnaloia closed 9 years ago

donnaloia commented 9 years ago

I ask this because I get a memory error during import. Do 'import all' import every city from every country? I was hoping a workaround would be to only import united states information.

The other work arounds to the memory issue seem very complicated.

Please advise.

coderholic commented 9 years ago

Yes import all imports data from all countries. There's no a command line option to import data for a single country, but one thing that should work is to import all country first (with import countries), and then delete all countries but the US. If you then import regions and then cities it'll only load those from the US.

donnaloia commented 9 years ago

Okay thanks. To delete all countries except the US, can I do something like Countries.objects.all().exclude(name='United States").delete() ?

donnaloia commented 9 years ago

I wanted to double check first, because I'm scared to do the wrong thing and delete everything.

coderholic commented 9 years ago

Yes Countries.objects.all().exclude(name='United States").delete(), or the SQL query DELETE FROM cities_country WHERE code != 'US'

After you've run that, and then imported the other data you can see how many cities you have for each country (which should only show data for the US) with the following query:

SELECT country_id, count(1) FROM cities_city GROUP BY country_id

donnaloia commented 9 years ago

okay thank you so much. I imported regions and cities like you told me. Do I also need to run a seperate import for Postal codes?

coderholic commented 9 years ago

Yeah.

donnaloia commented 9 years ago

Last question, I did 'manage.py sqlclear cities' and then 'manage.py syncdb' to clear the table and start over again. Next I did '/manage.py cities --import=Country' just like before but it just says INFO-file up to date.

For some reason Django thinks this information is already populated in the database even though I dropped the cities table, deleting everything. Any advice on how to proceed?

coderholic commented 9 years ago

You need to do --force. I think it's up to date because it sees the downloaded files.

On 17 February 2015 at 14:09, Benjamin Donnaloia notifications@github.com wrote:

Last question, I did 'manage.py sqlclear cities' and then 'manage.py syncdb' to clear the table and start over again. Next I did '/manage.py cities --import=Country' just like before but it just says INFO-file up to date.

For some reason Django thinks this information is already populated in the database even though I dropped the cities table, deleting everything. Any advice on how to proceed?

— Reply to this email directly or view it on GitHub https://github.com/coderholic/django-cities/issues/74#issuecomment-74765244 .

donnaloia commented 9 years ago

I can now see countries has been populated in the database, but when i query PostalCode.objects.all() it returns an empty list... I have used the --force --import=PostalCode and can see the information loading in the console window, it takes about 10-15 minutes, but after, PostalCode.objects.all() still returns an empty list and all of my PostalCode queries that previously worked, now do not work because there is no PostalCode data. Do you have any suggestions? I already did sqlclear cities and tried to rebuild the tables and repopulate database by doing sqlclear and then syncdb.

coderholic commented 9 years ago

Did you import regions/cities/districts first?

donnaloia commented 9 years ago

No, what I have previously done is import country, import city and import postalcode (in this order), and I had no problem. I did the same order again after I did a 'sqlclear cities', and now there is some issues. should I do import in a different order than this?

coderholic commented 9 years ago

That order should be fine. Can you confirm that the city import worked, and you now have cities?

donnaloia commented 9 years ago

Yes, I can confirm city import worked.

coderholic commented 9 years ago

And you have a CITIES_POSTAL_CODES setting?

donnaloia commented 9 years ago

Yes, I do.

donnaloia commented 9 years ago

It's very strange because when I import PostalCode I can see the activity in the console window, it appears to be loading the information, but still there is nothing, I'm so confused.

coderholic commented 9 years ago

What do you see when you do a select count(1) from cities_postalcodes from dbshell?

On 17 February 2015 at 15:47, Benjamin Donnaloia notifications@github.com wrote:

It's very strange because when I import PostalCode I can see the activity in the console window, it appears to be loading the information, but still there is nothing, I'm so confused.

— Reply to this email directly or view it on GitHub https://github.com/coderholic/django-cities/issues/74#issuecomment-74781136 .

donnaloia commented 9 years ago

Nothing happens, it just returns to the next line. This is my first time using dbshell, so maybe I am doing something incorrect. I typed 'select count(1) from cities_postalcodes' and hit return.

coderholic commented 9 years ago

you need to add a semicolon on the end, eg:

# select count(1) from cities_postalcode;
count
-------
91307
(1 row)
donnaloia commented 9 years ago

count


0 (1 row)

coderholic commented 9 years ago

Can you do the same for cities and countries?

On 17 February 2015 at 17:14, Benjamin Donnaloia notifications@github.com wrote:

count

0 (1 row)

— Reply to this email directly or view it on GitHub https://github.com/coderholic/django-cities/issues/74#issuecomment-74792033 .

donnaloia commented 9 years ago

count 250 for country. count 140926 for cities.

donnaloia commented 9 years ago

Should I try 'sqlclear cities' again?

coderholic commented 9 years ago

Sure you could give it a try. Might also be worth trying with a completely new DB.

donnaloia commented 9 years ago

Well, after doing sqlclear and then force import (for the fourth time), I can now see PostalCodes, I'm not sure what the problem was before, but I don't even want to think about it anymore.