implement validator for company existion

bundle exec rake company_status:check_all -- --open_data_file_path=lib/tasks/data/ettevotja_rekvisiidid__lihtandmed.csv --missing_companies_output_path=lib/tasks/data/missing_companies_in_business_registry.csv --deleted_companies_output_path=lib/tasks/data/deleted_companies_from_business_registry.csv --download_path=https://avaandmed.ariregister.rik.ee/sites/default/files/avaandmed/ettevotja_rekvisiidid__lihtandmed.csv.zip

This rake task performs the following actions:

downloads an archive
unzips it
checks all companies from our registry to see if they are in the business registry based on the downloaded data
if not present, a query is made to the business registry
if a company has been deleted, it is saved in the file specified here at deleted_companies_output_path, if information about the company is missing, it is saved in the file specified here at missing_companies_output_path
we set company status and validation date to the Contact model

Therefore, the attributes look like this:

open_data_file_path- specifies where the data is saved and retrieved from. Default value lib/tasks/data/ettevotja_rekvisiidid__lihtandmed.csv
missing_companies_output_path - specifies the path where companies not found in the business registry will be saved. Default value lib/tasks/data/missing_companies_in_business_registry.csv
deleted_companies_output_path - specifies the path where companies that have been removed from the registry will be saved. Default value deleted_companies_from_business_registry.csv
download_path - specifies where the data will be downloaded from. Default value https://avaandmed.ariregister.rik.ee/sites/default/files/avaandmed/ettevotja_rekvisiidid__lihtandmed.csv.zip

Since this command already includes default values, it is not necessary to enter any parameters; they were simply added for greater flexibility. Therefore, you can run the following command: bundle exec rake company_status:check_all

and the data will be available in the directory lib/tasks/data

The job: CompanyRegisterStatusJob.perform_later(days_interval = 14, spam_time_delay = 0.2, batch_size = 100, download_open_data_file_url='https://avaandmed.ariregister.rik.ee/sites/default/files/avaandmed/ettevotja_rekvisiidid__lihtandmed.csv.zip')

This job accepts the following parameters:

days_interval - selects domains that were last checked more than {days_interval} days ago.
spam_time_delay - this is the time delay when querying the business registry.
batch_size - the size of the batch for processing. This is needed for optimization.
download_open_data_file_url - the URL from which to download the business registry data.

As indicated above, all these values have default settings, so they can be modified if necessary.

What the job does:

It selects companies from Estonia that were checked N days ago or companies that are in liquidation/bankruptcy/removed from the registry - or generally contain no information about having been validated (NULL value).
For each of these, a request is made to the registry to determine the status.
If the status is K/N or there is no information, we set ForceDelete if it is not already set.
If the previous status was R, and the status in the business registry is R, we simply update the date of the check.
If a domain has ForceDelete due to the company's status, and the status is K/N, but the business registry shows status R, we cancel ForceDelete.
For domains in status_notes, we specify the following information Company no: {ident_number} if we set ForceDelete due to bankruptcy, company removal from the registry, or its absence.
If the domain status is L, we send them an email.

POTENTIAL PROBLEM: It could happen that we decide to check a large array of data in one day, and say the next time we decide to check in a year, and logically this job might process a large list of companies exactly one year later. This should be kept in mind.

this PR related to this one #https://github.com/internetee/company_register/pull/6

internetee / registry

implement validator for company existion #2601