nnjeim / world

A Laravel package which provides a list of the countries, states, cities, currencies, timezones and languages.
MIT License
748 stars 105 forks source link

Question about the SeedAction #30

Closed OzanKurt closed 2 years ago

OzanKurt commented 2 years ago

Is there a reason that you removed the fields before you seeded the database?

$this->forgetFields($countryFields, ['id']);

I find this completed redundant since the ids in the json resource match already.

OzanKurt commented 2 years ago

I've updated the seed to only take like ~15 seconds. Have a look, once you're okay with this you can update and release.

    public function run(): void
    {
        $this->command->getOutput()->block('Seeding start');

        // countries / timezones / currencies - START
        $countriesToInsert = [];
        $timezonesToInsert = [];
        $currenciesToInsert = [];
        $countryFields = Schema::getColumnListing(config('world.migrations.countries.table_name'));
        $currencyFields = ['name', 'code', 'precision', 'symbol', 'symbol_native', 'symbol_first', 'decimal_mark', 'thousands_separator'];

        foreach ($this->countries['data'] as $countryArray) {
            $countryArray = array_map(fn ($field) => gettype($field) === 'string' ? trim($field) : $field, $countryArray);

            $countriesToInsert[] = Arr::only($countryArray, $countryFields);

            foreach ($countryArray['timezones'] as $timezoneArray) {
                $timezonesToInsert[] = [
                    'country_id' => $countryArray['id'],
                    'name' => $timezoneArray['zoneName'],
                ];
            }

            $currencyDefaults = [
                'precision' => 2,
                'symbol_first' => 1,
                'decimal_mark' => '.',
                'thousands_separator' => ',',
                'country_id' => $countryArray['id'],
            ];

            if (array_key_exists($countryArray['currency'], $this->modules['currencies']['data'])) {
                $currency = $this->modules['currencies']['data'][$countryArray['currency']];

                $currencyArray = Arr::only($currency, $currencyFields);
            } else {
                $currencyArray = [
                    'name' => (string) $countryArray['currency'],
                    'code' => (string) $countryArray['currency'],
                    'symbol' => (string) $countryArray['currency_symbol'],
                    'symbol_native' => (string) $countryArray['currency_symbol'],
                ];
            }

            $currenciesToInsert[] = $currencyArray + $currencyDefaults;
        }
        // countries - END

        // states - START
        $statesToInsert = [];
        $stateFields = Schema::getColumnListing(config('world.migrations.states.table_name'));

        foreach ($this->modules['states']['data'] as $stateArray) {
            $stateArray = array_map(fn ($field) => gettype($field) === 'string' ? trim($field) : $field, $stateArray);

            $statesToInsert[] = Arr::only($stateArray, $stateFields);
        }
        // states - END

        // cities - START
        $citiesToInsert = [];
        $cityFields = Schema::getColumnListing(config('world.migrations.cities.table_name'));

        foreach ($this->modules['cities']['data'] as $cityArray) {
            $cityArray = array_map(fn ($field) => gettype($field) === 'string' ? trim($field) : $field, $cityArray);

            $citiesToInsert[] = Arr::only($cityArray, $cityFields);
        }
        // cities - END

        // languages - START
        $languagesToInsert = $this->modules['languages']['data'];
        // languages - END

        /**
         * START -> Inserting All Models
         */

        // Calculate how many queries we will be running
        $citiesToInsertChunks = array_chunk($citiesToInsert, 1000);

        $this->command->getOutput()->progressStart(4 + count($citiesToInsertChunks));

        Models\Country::insert($countriesToInsert);
        $this->command->getOutput()->progressAdvance();

        Models\Timezone::insert($timezonesToInsert);
        $this->command->getOutput()->progressAdvance();

        Models\Currency::insert($currenciesToInsert);
        $this->command->getOutput()->progressAdvance();

        Models\State::insert($statesToInsert);
        $this->command->getOutput()->progressAdvance();

        foreach ($citiesToInsertChunks as $citiesToInsertChunk) {
            Models\City::insert($citiesToInsertChunk);
            $this->command->getOutput()->progressAdvance();
        }

        Models\Language::insert($languagesToInsert);
        $this->command->getOutput()->progressAdvance();
        /**
         * END -> Inserting All Models
         */

        $this->command->getOutput()->progressFinish();

        $this->command->getOutput()->block('Seeding end');
    }
sakon54321 commented 2 years ago

Wow @OzanKurt , this works amazing and super fast!!! Well done.

nnjeim commented 2 years ago

Hi @OzanKurt Please submit your code in a separate branch for us to test. I really appreciate your contribution. Following this will create a pull request. 1 comment here. the conditional seeding coming the config file are they still respected?

nnjeim commented 2 years ago

Is there a reason that you removed the fields before you seeded the database?

$this->forgetFields($countryFields, ['id']);

I find this completed redundant since the ids in the json resource match already.

I believe the first seed (empty tables never caused an issue), when re-seeding some contributors added this to avoid db issues.

nnjeim commented 2 years ago

close for the lack of activity

sakon54321 commented 1 year ago

Hi @nnjeim , is this fix getting incorporated into the package? It seems to be the only thing that makes it load past 40% for me and in only a fraction of the time.

nnjeim commented 1 year ago

Hi @sakon54321 the seed action requires you to set memory_limit = -1 in php.ini try this alternative please. let me know if it works for you.

sakon54321 commented 1 year ago

I'm not sure I'm doing it correctly because it still gives me problems @nnjeim. But the fix works perfectly and quickly every time.