Migrate Countries and Regions to External Dataset

[SylencerWebdesign]

Hi

Great project, thanks mate, keep going!

For me, in the World Travel Section I have only 21 Countries for Selection. Ist it possible to add all countries?

I followed the guide and did this already:

python manage.py worldtravel-seed --force

It run through, but only for the 21 countries. What about all other contries? Any advice?

[seanmorley15]

Hi!, The World Travel section only has 21 countries because it can be time consuming to add all of them, I have only added 21 so far but would like to add more. This file is what is used to insert the world travel data into the DB. The region codes come from ISO standards. Here is where you can search by country code for regions https://www.iso.org/obp/ui/#iso:code:3166:US (replace the last 2 with desired country code) It would be great if people who are knowledgeable about a certain country contribute to it (little to no coding knowledge required)! Thanks!

[lovwyr]

Just a Question, do you plan to extend this feature more in the "Deep"? Like adding counties or Cities and then allow linking adventures/Collections again any level?

[seanmorley15]

Hmm. I see what you are saying, I will think about how this feature can be extended while still keeping it easy to use.

[lovwyr]

btw. while looking for the CZ Data my Brain produced an additional Idea 😊

how about splitting the region names into local and international (eng) Labels, some of the local names can become an "hurdle" for non-locals (Greek and Finnland for Example)

[seanmorley15]

Great idea! I will look into this

[seanmorley15]

@lovwyr, how does this look. Each region now has a name_en field (some regions are hard to translate / don't fit well translated so those stay the same)

[lovwyr]

Looks good :), i think for the moment its fine to stay with this Solution.

[dymek37]

Hi, If you still need help, I will prepare a list of all States over the weekend and then start preparing the regions. How can I send the list to you? Just a txt file with the fragment countries = [...] ?

[seanmorley15]

Hi, Thanks for wanting to help! What would be really useful is taking a look at this file that has all of the information for the world travel data and adding any countries and regions you are interested in. Countries and region info can be found here: https://www.iso.org/obp/ui/#iso:code:3166:US (replace the last 2 with desired country code)

[dymek37]

Hi, I added 249 countries in alphabetical order (almost ;)). I created a pull request. I was doing it for the first time and I don't know if I did everything correctly.

[dymek37]

Hi, @seanmorley15 Can you tell me if I did it the right way? I'd like to start uploading regions for countries, but I want to make sure I'm doing it right. Thank you.

[seanmorley15]

@dymek37 Thanks so much for adding all of the countries! If you want to add regions based on the ISO criteria to these countries that would be great! If you could comment out the countries that do not have any regions (because its a lot of work) that would be great so people can add regions to them in the future. This is because only regions are marked as visited and not the countries. (I just merged it into main) Thanks so much for your help!

[dymek37]

@seanmorley15 You're welcome 😊 I'm glad I could help. I'll start adding regions for all countries now, probably alphabetically, as that seems like the easiest way. I'll also try to comment out the countries without regions. I should be able to upload the first batch of regions today. P.S. I really like your app! I travel a lot, and I’ve been missing something like this. Great job 😊

[seanmorley15]

@dymek37 Thanks so much for the help! I just commented out the countries without regions in the main branch. Anyone can uncomment a country, add regions and then open a pull request! Thanks so much and I'm glad you like the app!

[dymek37]

@seanmorley15 The application is brilliant :) One more question: if a country doesn't have a region, should I add the same country code in the region as in the country?

[seanmorley15]

@dymek37 I never thought of that before! Yes, that would be a good idea to just add the country as the single region. Thanks!

[tigattack]

Hi @seanmorley15 - Firstly I just want to say I've not been using it for long, but I'm really enjoying AdventureLog so far! You've done an awesome job with it.

I was wondering if you'd considered using an existing dataset for the regions and countries? There are tons of great resources that could potentially save you and other contributors time from manually maintaining ~3500 lines of country/region data (which will only grow).

Using an existing and complete dataset could free up time to focus on other parts of AdventureLog or perhaps even contribute upstream, which could benefit the broader community.

A few examples that could be worth exploring:

• A library: https://github.com/amckenna41/iso3166-2 This is not a complete solution as it cannot replace the countries list, but IMO a great demonstration of how >3000 lines of regions could be replaced with 6:

  from iso3166_2 import ISO3166_2
  iso = ISO3166_2()
  # Assuming `countries` is already defined:
  regions = [
      (region_code, region_info['name'], region_info['localName'], country_code)
      for _, country_code, _ in countries
      for region_code, region_info in iso.all.get(country_code.upper(), {}).items()
  ]

  The result of this is exactly the same schema as the existing regions list, but without the maintenance 😉 A nice bonus over the existing data in this repository is it also returns lat-long for each region, which could help with visualising a map of visited locations for example. If a library approach is preferred, something such as pycountry could be combined with this library to provide the full set of information required.

• A dataset: https://github.com/dr5hn/countries-states-cities-database is a project that maintains and distributes various geo-metadata in a pretty comprehensive list of formats. Based on some (very quick) research, this would probably be my choice since it contains all the data already encapsulated here (and much more!) and can be used in whatever way you please due to the number of different distribution formats. This could be pulled on start and adding to the DB, used in JSON, etc...

• An API: https://continentl.com/docs/world-global-country-api allows querying countries and their regions.

I do appreciate that having both the localised and Anglicised region names in the existing dataset is a nice feature, however, given the effort required to maintain a custom list, it might be worth considering one of these options.

I'd be more than happy to contribute a PR integrating one of these solutions, but I thought it would be best to discuss first and hear your thoughts.

[seanmorley15]

Wow @tigattack! This is exactly what I needed thank you so much! I totally agree with all of it. Once I get this sharing feature update finished I will work on migrating to the dataset found here. I want to thank @dymek37 so much for the hard work making the current system so much better!Although this hard work is not sustainable for the future and this dataset is ultimately the best choice for consistent data and scalability. I can take charge of this next because I will make it use the dataset to create db objects that are compatible with the current system. I am super excited for this new dataset because the features and possibilities are so much greater!!! I truly appreciate everyone helping to make AdventureLog the best it can be!

[tigattack]

Hey that's great! I'm glad I could help. Looking forward to seeing the results!

[dymek37]

@seanmorley15 I'm glad that come this way I could help 🙂

Continue to add the remaining administrative subdivisions for the missing countries? Or is it no longer necessary due to the planned changes?

[seanmorley15]

@dymek37, I would hold off on adding the rest of the regions. I don't want you to have to spend time doing something that will be updated soon. Once again, thanks so much for all the help :)

[dymek37]

@seanmorley15 I'm glad I could help :) I'm looking forward to the next updates :)

[seanmorley15]

Hi everyone! I am excited to announce that the migration to the external dataset has been mostly completed and is a huge success. The import time for countries and regions has been decreased from about 30-60secs to about 0.5secs and with over 5000 regions and 250 countries. This feature is also backwards compatible with the old one so visited regions will remail the same. If you want you can manually build the docker images by commenting out the image and uncommenting the build command on the worldtravel-v2 branch. Although, this update its planned for general release before the end of the week! Thanks so much everyone for all the help!

Edit: also the old command is still present for now but the new one is download-countries

[tigattack]

Looks great! 🎉

One suggestion: On this line https://github.com/seanmorley15/AdventureLog/blob/worldtravel-v2/backend/server/worldtravel/management/commands/download-countries.py#L41 I would recommend pinning to a released version tag. It adds some manual effort for keeping it up to date, but it's also slightly safer since avoids pulling any accidental/malicious bad updates on the master branch directly into AdventureLog users' DBs.

Edit: This was addressed by 6ac3f0541f9ea3ec1072358aa594843c049f2500.