Import Location Test1 - Not working in a localhost installation

[betovicentini]

• I tested importing administrative boundaries into OpenDataBio, as a first step to import the duckewiki stored data.
• It imported into http://opendatabio.ib.usp.br/opendatabio fine.
• But it did not work when importing into my localhost installation. I can get data like odb_get_taxons(), but cannot import locations. All my attempts failed. The jobs do not conclude and stay on hold.
• Commands submitted where as follow:

  library(raster)
  lv = 0
  ff = 0
  while(ff==0) {
  ocrt <- try(getData('GADM', country=crt$Codes[t], level=lv),silent = T)
  if (class(ocrt)=="try-error") {ff=1} else {
    odb_import_locations(sp_to_df(ocrt), odb_config(base_url=base_url, token = token))
    lv = lv+1
  }
  }

• Notes: -- I've successfully imported data for country level of Peru twice. Hence, import script is not checking if location is already in database;

[birutaibm]

What was the error message when you in your computer? I try import some locations in my computer, and it does not work, too. My server returns HTTP error 413 (Request Entity Too Large). I create a .htaccess at my opendatabio instalation directory with the content: php_value post_max_size 100M php_value upload_max_filesize 100M But it still does not work.

[EDIT: re-posted as #13]

[betovicentini]

Restating. It worked and submitted the Job. However, the Job, which should import the data is not working. I don't get an error message. Just the job does not conclude after submission. Change in php.ini and the restart apache. Try a smaller file (country level=0).

[andrechalom]

Hello, Beto.

There are some different issues here. The first is probably due to supervisor / queue management configuration.

Please check the following:

• Is supervisor installed and running? Is the supervisor configuration file correct? This may require some tweaking in systems other than Debian/Ubuntu or Archlinux, and I´m not able to help here.
• Check if the job is being tried and abandoned or if supervisor does not even attempt it. There are two ways of checking this: open the supervisor log file (should be in storage/logs/supervisor.log). If it does not exist or is empty, this means supervisor is not starting the job. The second way is checking the database, in the table "jobs", such as "SELECT id, queue, attempts FROM jobs". If there are jobs in this table with attempts = 0, this means supervisor is not starting them. If there are attempts > 0, it means supervisor is attempting them but failing. If there are no jobs in this table, it proably means that the jobs are being processed.

An alternative way of processing jobs is

• Stop supervisor.
• Open a terminal and change directory to the root odb installation
• run php artisan queue:listen and check the output.

The other issues are related to "post max size" and data duplication when importing locations from Peru. These are not related, please open different issues to track possible bugs for those. Also please add more details to help identify the problem. What is the size of the data that you are trying to import? Can you provide a minimal example of data being duplicated?

On Fri, Jan 26, 2018 at 4:17 PM, Alberto Vicentini <notifications@github.com

wrote:

Restating. It worked and submitted the Job. However, the Job, which should import the data is not working. I don't get an error message. Just the job does not conclude after submission. Change in php.ini and the restart apache. Try a smaller file (country level=0).

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/opendatabio/opendatabio-r/issues/11#issuecomment-360862611, or mute the thread https://github.com/notifications/unsubscribe-auth/AB5w9e_lHmXbE-OYF7kUA0H_RdPp5qx5ks5tOhbRgaJpZM4RsR_u .

[andrechalom]

Regarding the data duplication, please confirm that in your local installation the unique key is correctly set on the mysql database (you can check this by entering the mysql console and typing show create table locations):

 UNIQUE KEY `locations_name_parent_id_unique` (`name`,`parent_id`),

[betovicentini]

The problem was OpenDataBio not finding the supervisor configuration file, because it is not placed in a Mac in the same path (usually /usr/local/etc/). I did add to the MacInstructions how to config supervisor to make it running. Now is working fine. I had to create some folders manually, including the app/storage and app/storage/log within the opendatabio installation.

[betovicentini]

Closing this issue replacing with #14