Public portal doesn't work, HTTP 50x returned by backend service, or unexpected value (HTTP 200, no json value)

[TomasKulhanek]

I got HTTP 504 Gateway Time-out on https://portal.west-life.eu/metadataservice/files Which seems that the container with backend service (.NET Metadataservice) is not working. Can you check it, please.

[TomasKulhanek]

fixed

[TomasKulhanek]

Several issues discovered.

• [x] docker container on public portal (update based on commit on github) appears to be hard to maintain. Switched to VM (update distributed via cvmfs) - from appdb.egi.eu (recently the vagrant box), using binaries from /cvmfs/west-life.egi.eu/software/virtualfolder. Migration of data in process (Francois)
• [x] during migration, database with encrypted user settings needs to be migrated using old and new private keys. (Francois)
• [x] backend service fails on Segmentation fault - using distribution Mono 4.6.2, tested on prebuilt mono 4.6.1, the same problem is reproducible (Tomas)
• [x] VRE service returns unexpected result on error during getting authproxy info of sessionid pz3702xm0skrq7n1ynql9qf5qk4livnf domain portal.west-life.eu” is thrown when the VRE api fails to return webdav proxy to metadataservice . (Tomas,Francois)

[TomasKulhanek]

Nov 29 20:59:46 localhost mono: error during getting user info of sessionid pz3702xm0skrq7n1ynql9qf5qk4livnf Could not load file or assembly or one of its dependencies.  at ServiceStack.ServiceClient.Web.JsonServiceClient.DeserializeFromStream[T] (System.IO.Stream stream) [0x00000] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: at ServiceStack.ServiceClient.Web.ServiceClientBase.HandleResponse[TResponse] (System.Net.WebResponse webResponse) [0x000bd] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: at ServiceStack.ServiceClient.Web.ServiceClientBase.Send[TResponse] (System.String httpMethod, System.String relativeOrAbsoluteUrl, System.Object request) [0x00056] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: error during getting authproxy info of sessionid pz3702xm0skrq7n1ynql9qf5qk4livnf domain portal.west-life.eu
Nov 29 20:59:46 localhost mono: UNAUTHORIZED  at ServiceStack.ServiceClient.Web.ServiceClientBase.ThrowWebServiceException[TResponse] (System.Exception ex, System.String requestUri) [0x00144] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: at ServiceStack.ServiceClient.Web.ServiceClientBase.ThrowResponseTypeException[TResponse] (System.Object request, System.Exception ex, System.String requestUri) [0x00003] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: at ServiceStack.ServiceClient.Web.ServiceClientBase.HandleResponseException[TResponse] (System.Exception ex, System.Object request, System.String requestUri, System.Func`1[TResult] createWebRequest, System.Func`2[T,TResult] getResponse, TResponse& response) [0x000cb] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: at ServiceStack.ServiceClient.Web.ServiceClientBase.Send[TResponse] (System.String httpMethod, System.String relativeOrAbsoluteUrl, System.Object request) [0x00062] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: at ServiceStack.ServiceClient.Web.ServiceClientBase.Get[TResponse] (System.String relativeOrAbsoluteUrl) [0x00000] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: at MetadataService.Services.Files.VreCookieRequestFilterAttribute.GetAuthProxy (System.String sessionid, System.String domain) [0x00052] in <241aefa1a1f1418fbafc180ef4bcae37>:0
Nov 29 20:59:46 localhost mono: error during getting user info of sessionid pz3702xm0skrq7n1ynql9qf5qk4livnf Could not load file or assembly or one of its dependencies.  at ServiceStack.ServiceClient.Web.JsonServiceClient.DeserializeFromStream[T] (System.IO.Stream stream) [0x00000] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: at ServiceStack.ServiceClient.Web.ServiceClientBase.HandleResponse[TResponse] (System.Net.WebResponse webResponse) [0x000bd] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: at ServiceStack.ServiceClient.Web.ServiceClientBase.Send[TResponse] (System.String httpMethod, System.String relativeOrAbsoluteUrl, System.Object request) [0x00056] in <3f1d9d393843495b897d989976fba7e7>:0
Nov 29 20:59:46 localhost mono: Stacktrace:
Nov 29 20:59:46 localhost mono: Native stacktrace:
...

log.txt

[TomasKulhanek]

SIGSEGV solved, this is due to some I/O issues of booted cernvm – I switched the mono to the one I have at /cvmfs/west-life.egi.eu/tools/mono and the service starts working.

There are some other I/O issues on the VM

• davfs2 can’t be mounted due to it, thus B2DROP fails to show some content.
• DROPBOX can’t be set due to sqlite exception – can’t open the file. rebooting the VM seems to fix it for some time

[TomasKulhanek]

Public portal now operable and working:

• portal backend server was reinstalled and migrated to new deployment architecture
• host/guest configuration instability prevented to be operable immediately last week, solved by upgrade of hypervisor.
• unfortunately encrypted db with user's data was lost, thus after login you need to register your connection to b2drop, dropbox and other services again.