Upgrade ANDS R11 nectar image to R13

sunyi000 commented 10 years ago

I followed the instructions here to create a new VM with ANDS netctar image.

http://community.ands.org.au/viewtopic.php?f=211&t=3271

and then upgraded it to R13 based on the R12, R13 installation notes on Github. After all the installations, I got the following issues

I logged into the registry and go "Manage my data source" but the page is blank.
if I click ‘setup an example datasource with a few sample records’ from ‘My registry Home’, it shows an error in /var/www/html/application/registry/registry_object/models/extensions/transforms.php on line 138, which is the line below $TimePeriodList = $sxml-xpath(‘//TimeperiodList’)[0]
I checked the apache error log, there're other two errors [Tue Aug 05 11:11:02 2014] [error] [client 152.98.200.196] PHP Parse error: syntax error, unexpected '[' in /var/www/html/applications/registry/services/controllers/rda.php on line 475, referer: http://130.102.154.246/ [Tue Aug 05 11:11:09 2014] [error] [client 130.102.154.246] Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.

I have no idea about the first apache error. The 2nd apache error seems related to the .htaccess file in ANDS R11 image. Could anyone please help?

Thanks

minhd commented 10 years ago

The .htaccess file should be identical to https://github.com/au-research/ANDS-Registry-Core/blob/master/htaccess.sample Could you post your .htaccess file or is there any vhost configuration that is doing the recursion?

sunyi000 commented 10 years ago

I dont have any vhost config. The error appears straight away after I launched the nectar image and 'yum update'

DirectoryIndex index.php

RewriteEngine on RewriteBase /

RewriteRule ^services/ registry/ [R=301,L]

RewriteCond $1 ^(css|img|js|less|lib|uploads|shared) RewriteRule ^assets/(.?)/(.)$ assets/$1/$2 [L] RewriteRule ^assets/(.?)/(.)$ applications/portal/$1/assets/$2 [L]

RewriteCond $1 !^applications/./ RewriteRule ^(.?)/assets/(.?)/(.)$ applications/$1/$2/assets/$3 [L]

RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUESTURI} !(.)/$ RewriteRule ^(._)$ $1/ [L]

RewriteCond $1 !^(index.php|robots.txt|favicon.ico|legacy|googledocverificationidfile.html) RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUESTFILENAME} !-d RewriteRule ^(.?)/(._)$ ./index.php?/$2&app=$1 [L,QSA]

sunyi000 commented 10 years ago

What I did was, installed from ANDS netcar image and then run through all the upgrade steps I could find on Github (git fetch, sql scripts etc) and installed the new harvester.

bgreenwood commented 10 years ago

The first issue is related to PHP version. Dereferencing a method call directly is only possible from PHP 5.4. Suggest fixing this code to store in temporary variable before dereferencing: $tmp = getArray(); $secondElement = $tmp[1];

http://php.net/manual/en/language.types.array.php#example-102

On Tue Aug 05 2014 at 2:10:25 PM sunyi000 notifications@github.com wrote:

I dont have any vhost config. The error appears straight away after I launched the nectar image and 'yum update'

DirectoryIndex index.php

RewriteEngine on RewriteBase /

RewriteRule ^services/ registry/ [R=301,L]

RewriteCond $1 ^(css|img|js|less|lib|uploads|shared) RewriteRule ^assets/(.?)/(.)$ assets/$1/$2 [L] RewriteRule ^assets/(.?)/(.)$ applications/portal/$1/assets/$2 [L]

RewriteCond $1 ^(registry|roles|apps|developer) RewriteCond $2 ^(css|img|js|less|lib) RewriteRule ^(.?)/assets/(.?)/(.*)$ assets/$2/$3 [L]

RewriteCond $1 !^applications/. / RewriteRule ^(.?)/assets/(.?)/(.)$ applications/$1/$2/assets/$3 [L]

RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUESTURI} !(. )/$ RewriteRule ^(._)$ $1/ [L]

RewriteCond $1 !^(index.php|robots.txt|favicon.ico|legacy|googledocverificationidfile.html) RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUESTFILENAME} !-d RewriteRule ^(.?)/(._)$ ./index.php?/$2&app=$1 [L,QSA]

— Reply to this email directly or view it on GitHub https://github.com/au-research/ANDS-Registry-Core/issues/67#issuecomment-51147871 .

sunyi000 commented 10 years ago

I think the recursion error was caused by '/assets', in .htaccess file. It is redirected from /assets/shared/dashboard_news/dashboard.html
is dashboard_news a folder under /assets/shared or a config somewhere? there're only two folders, 'topic' and 'spotlight' in /assets/shared

I commented out the assets part in .htaccess file for now

minhd commented 10 years ago

Yes, it's where we normally put the news article for the dashboard. You can create a folder dashboard_news under assets/shared and have a file called dashboard.html there

sunyi000 commented 10 years ago

regarding the php version issue, I fixed it on line 138 of transform.php but the same issue appears at other lines as well. Is that possible to let me know all the places where I need to fix? or please could you update those files which possibly contains the issue so I can just git pull? Do you suggest I should upgrade php to 5.4 or later? Thanks very much

minhd commented 10 years ago

Yes, it would be best to upgrade to PHP 5.4, efforts will be put in to determining the cause and make it compatible with older versions of PHP.

minhd commented 10 years ago

To assist with the issue, a NecTAR image has been created for R13, available publicly as ANDS-MetadataRegistry-v13-Test https://dashboard.rc.nectar.org.au/project/images_and_snapshots/23580597-72c1-4153-9f16-372f33cfd0c0/

Please have a look with this new image and let me know

sunyi000 commented 10 years ago

Thanks Minh,I will try the image and let you know

minhd commented 10 years ago

Please note that the image is still in Testing mode. To get it working on your particular server, you might have to change some configurations in the global_config.php file

sunyi000 commented 10 years ago

couple of issues i have so far after trying the image

Steps I have done:

launched R13 image from nectar
modified global_config.php (changed default_base_url to our server ip)

Issues:

the same "Request exceeded the limit" error in httpd log.
tried to create a testing data source, after clicking the "Save" button, it shows "Saving..." forever
on Harvester Settings page, nothing in 'Harvest Method' dropdown

Questions and clarifications needed so far:

The harvester and solr were installed in the image.
what is the equivalent provider type to the old 'RIF OAI-PMH' in ORCA8

Thanks

sunyi000 commented 10 years ago

Issue 1 was solved by creating a directory in /assets/shared/dashboard_news/

minhd commented 10 years ago

The request exceed the limit error is currently being looked at. We'll have a patch for it sooner or later.

Yes, the harvester and SOLR were supposedly installed in the image, however we ran into a couple of small problem with the harvester not starting as expected, this is also currently being looked at.

The harvester will provide the required harvest method and provider types for the registry, and RIFCS OAI will also be available as soon as the harvester is online in the image.

minhd commented 10 years ago

The second release candidate for the R13 snapshot is now available as: ANDS-MetadataRegistry-v13-Test-RC2 https://dashboard.rc.nectar.org.au/project/images_and_snapshots/e870a222-4259-46c0-bf82-9ed0d5fc50c7/

Sissvoc server preloaded with ANDS required vocabularies and harvester are now fully working with this image.

sunyi000 commented 10 years ago

Hi Minh, I launched the new image and created a data source to test but had an error when click "Save". can you please help. The error trace is as below

sunyi000 commented 10 years ago

I selected 'Do not have contributor pages' in edit settings page. Tried debug the ds_app.js code, $scope.ds.contributor.items is empty on line 224.

Thanks

minhd commented 10 years ago

It seems to be a bug, try auto generate a contributor page for the data source. I'll get around and fix this problem as soon as I'm able.

sunyi000 commented 10 years ago

same error. Should it be institution_pages instead of contributor_pages ?

minhd commented 10 years ago

What happens if you unset the contributor page value from the database? try deleting the field in data_source_attributes where attribute is institution_pages

minhd commented 10 years ago

Could you cherry pick this commit https://github.com/au-research/ANDS-Registry-Core/commit/8050991bd115378b317d0ad23d62d1216629440d I've just done a quick fix and put it to our develop branch waiting to be deployed for R13.1

sunyi000 commented 10 years ago

That eliminates the error, but the data source name is not displayed on 'Manage my data source' page

minhd commented 10 years ago

Can you check the value for the datasource title in the database? What's the value in the edit settings screen?

sunyi000 commented 10 years ago

Thanks Minh, 2 other issues

1.Does the button "Import from Harvester" initiate harvest? Nothing happens after clicking the button, i could see from the log 'harvest scheduled to run at xxxxxxxx' but the number of records harvested is 0.

another error on harvester dashboard page appears after I create the new data source

minhd commented 10 years ago

Hi Yi

Could you please check if the harvester is running with ps -ef | grep harvester. If it's not running then can you start it?

There should be another image ANDS Registry R13 Test RC 3 that we released yesterday with some more bugfixes, hope this will help

sunyi000 commented 10 years ago

it's running. actually to make sure harvester is running I did ./startHarvester.sh every time before I create a data source. Is it ok to do so? Thanks

minhd commented 10 years ago

It will create multiple instance of harvester, that will maybe slow the system down, but should be functioning.

The way the harvester works is that it reads from the harvests table every interval and do it's harvest thing, then tell the importer to start importing. Can you please check if there's any entry in the harvests table? there should be your datasource with the status of SCHEDULED

minhd commented 10 years ago

Could you please try out the ANDS-MetadataRegistry-v13-Test-RC3 image? we implemented a few fixes and incorporated that into this version.

Also, you can do git pull on the repo to update to the latest version, if required

sunyi000 commented 10 years ago

There's one entry in the harvests table. From the log, all I get is the below message, and there's no records harvested. I go to the harvester dashboard and the above 'length' error appears

Harvest scheduled to run at 2014-08-14 09:50:19 URI: http://dap.ozflux.org.au/oai/provider Harvest Method: OAI-PMH Harvester Provider Type: rif Advanced Harvest Mode: STANDARD

sunyi000 commented 10 years ago

Can I just git pull on the RC2 image, instead of creating a new one?

minhd commented 10 years ago

have a try on doing a git pull on the harvester at /usr/local/harvester and another one at /var/www/core/.

I believe the only thing that is not in the master branch yet is the daemon implementation of the harvester

leo-monus commented 10 years ago

can you check the SYSTEM TIME? if the system is running at UTC and the harvest is scheduled in AEST. the harvest won't start for 10 hours

On 14/08/2014 10:56 am, Minh Duc Nguyen wrote:

It will create multiple instance of harvester, that will maybe slow the system down, but should be functioning.

The way the harvester works is that it reads from the |harvests| table every interval and do it's harvest thing, then tell the importer to start importing. Can you please check if there's any entry in the |harvests| table? there should be your datasource with the status of |SCHEDULED|

— Reply to this email directly or view it on GitHub https://github.com/au-research/ANDS-Registry-Core/issues/67#issuecomment-52131828.

sunyi000 commented 10 years ago

I did git pull at /usr/local/harvester and /var/www/core. System time is Thu Aug 14 11:17:04 EST 2014. checked harvester by doing ps -ef | grep harvester

still no records are harvested, and the same 'length' error in Harvester dashboard page

sunyi000 commented 10 years ago

I was logged out from ORCA and could not log back in again using superuser/superuser. I did not change anything in authentication_built_in table in dbs_roles.

I will try putting a new user in that table and log in.

minhd commented 10 years ago

The fix for the login issue is a script in the maintenance migrate_roles_to_r131 this basically changed all t value into 1 value. This will enable user login for the master branch.

au-research / ANDS-Registry-Core

Upgrade ANDS R11 nectar image to R13 #67