search

openlink / virtuoso-opensource

Virtuoso is a high-performance and scalable Multi-Model RDBMS, Data Integration Middleware, Linked Data Deployment, and HTTP Application Server Platform
https://vos.openlinksw.com
Other
855 stars 210 forks source link

HTTP response code: 503 #1029

Closed plegault3397 closed 2 years ago

plegault3397 commented 2 years ago

I keep getting this error. I can restart at the same place it fails and it goes through, then fails in another spot. Error: Unkown error Endpoint: /sparql HTTP response code: 503

TallTed commented 2 years ago

I'm sorry, but you've provided insufficient information for us to analyze and resolve this issue.

The more detail you can provide, including specific versions of all software involved, including but not limited to the Virtuoso server itself, and any client libraries or tools (ODBC, JDBC, ADO.NET driver/provider; web browser; etc.).

Best case, a step-by-step which consistently produces this 503 in your environment should let us reproduce it locally, or even reveal an immediate configuration change to resolve it for you.

plegault3397 commented 2 years ago

@TallTed Sorry should have added those details MediaWiki | 1.35.5 PHP | 7.4.15 Semantic MediaWiki | 4.0.1 Red Hat Enterprise Linux release 8.5 Virtuoso Version: 07.20.3233

Following their instruction here https://www.semantic-mediawiki.org/wiki/Help:SPARQLStore/RepositoryConnector/Virtuoso

I do not have any issues using the same with an earlier release

MediaWiki | 1.35.5 PHP | 7.4.15 Semantic MediaWiki | 4.0.1 Red Hat Enterprise Linux Server release 7.9 Virtuoso Version: 07.20.3230

The full crash output is:

SMW\SPARQLStore\Exception\BadHttpEndpointResponseException from line 83 of D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\SPARQLStore\HttpResponseErrorMapper.php: A SPARQL query error has occurred
Query: PREFIX wiki: <https://dvl-datamap.jnj.com/id/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX swivt: <http://semantic-mediawiki.org/swivt/1.0#>
PREFIX property: <https://dvl-datamap.jnj.com/id/Property-3A>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
SELECT DISTINCT ?result WHERE {
?result swivt:wikiPageSortKey ?resultsk .
?result property:Is_archived "no" .
?result property:Has_business_owner wiki:Epsilon .
 OPTIONAL { ?result swivt:redirectsTo ?o3 } .
 FILTER ( !bound( ?o3 ) ) .

}
ORDER BY ASC(?resultsk)
OFFSET 0
LIMIT 501
Error: Unkown error
Endpoint: http://awsacrnval0003.jnj.com/sparql
HTTP response code: 503

#0 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\SPARQLStore\HttpResponseErrorMapper.php(55): SMW\SPARQLStore\HttpResponseErrorMapper->createResponseToHttpError()
#1 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\SPARQLStore\RepositoryConnectors\GenericRepositoryConnector.php(642): SMW\SPARQLStore\HttpResponseErrorMapper->mapErrorResponse()
#2 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\SPARQLStore\RepositoryConnectors\GenericRepositoryConnector.php(497): SMW\SPARQLStore\RepositoryConnectors\GenericRepositoryConnector->mapHttpRequestError()
#3 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\SPARQLStore\RepositoryConnectors\GenericRepositoryConnector.php(215): SMW\SPARQLStore\RepositoryConnectors\GenericRepositoryConnector->doQuery()
#4 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\SPARQLStore\QueryEngine\QueryEngine.php(181): SMW\SPARQLStore\RepositoryConnectors\GenericRepositoryConnector->select()
#5 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\SPARQLStore\QueryEngine\QueryEngine.php(118): SMW\SPARQLStore\QueryEngine\QueryEngine->getInstanceQueryResult()
#6 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\SPARQLStore\SPARQLStore.php(310): SMW\SPARQLStore\QueryEngine\QueryEngine->getQueryResult()
#7 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\SPARQLStore\SPARQLStore.php(299): SMW\SPARQLStore\SPARQLStore->fetchQueryResult()
#8 D:\app\www\mediawiki\extensions\SemanticMediaWiki\includes\query\SMW_QueryProcessor.php(338): SMW\SPARQLStore\SPARQLStore->getQueryResult()
#9 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\ParserFunctions\AskParserFunction.php(364): SMWQueryProcessor::getResultFromQuery()
#10 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\ParserFunctions\AskParserFunction.php(196): SMW\ParserFunctions\AskParserFunction->doFetchResultsFromFunctionParameters()
#11 D:\app\www\mediawiki\extensions\SemanticMediaWiki\src\ParserFunctionFactory.php(402): SMW\ParserFunctions\AskParserFunction->parse()
#12 D:\app\www\mediawiki\includes\parser\Parser.php(3340): SMW\ParserFunctionFactory->SMW\{closure}()
#13 D:\app\www\mediawiki\includes\parser\Parser.php(3047): Parser->callParserFunction()
#14 D:\app\www\mediawiki\includes\parser\PPFrame_Hash.php(263): Parser->braceSubstitution()
#15 D:\app\www\mediawiki\includes\parser\Parser.php(3225): PPFrame_Hash->expand()
#16 D:\app\www\mediawiki\includes\parser\PPFrame_Hash.php(263): Parser->braceSubstitution()
#17 D:\app\www\mediawiki\includes\parser\Parser.php(3225): PPFrame_Hash->expand()
#18 D:\app\www\mediawiki\includes\parser\PPFrame_Hash.php(263): Parser->braceSubstitution()
#19 D:\app\www\mediawiki\includes\parser\Parser.php(2887): PPFrame_Hash->expand()
#20 D:\app\www\mediawiki\includes\parser\Parser.php(1556): Parser->replaceVariables()

I have not gotten a clean rebuilddata sense I have tried to switch to the new server, over the last week.

plegault3397 commented 2 years ago

virtuoso.ini:

;
;  virtuoso.ini
;
;  Configuration file for the OpenLink Virtuoso VDBMS Server
;
;  To learn more about this product, or any other product in our
;  portfolio, please check out our web site at:
;
;      http://virtuoso.openlinksw.com/
;
;  or contact us at:
;
;      general.information@openlinksw.com
;
;  If you have any technical questions, please contact our support
;  staff at:
;
;      technical.support@openlinksw.com
;

;
;  Database setup
;
[Database]
DatabaseFile                    = /app/virtuoso/var/lib/virtuoso/db/virtuoso.db
ErrorLogFile                    = /app/virtuoso/var/lib/virtuoso/db/virtuoso.log
LockFile                        = /app/virtuoso/var/lib/virtuoso/db/virtuoso.lck
TransactionFile                 = /app/virtuoso/var/lib/virtuoso/db/virtuoso.trx
xa_persistent_file              = /app/virtuoso/var/lib/virtuoso/db/virtuoso.pxa
ErrorLogLevel                   = 7
FileExtend                      = 200
MaxCheckpointRemap              = 2000
Striping                        = 0
TempStorage                     = TempDatabase

[TempDatabase]
DatabaseFile                    = /app/virtuoso/var/lib/virtuoso/db/virtuoso-temp.db
TransactionFile                 = /app/virtuoso/var/lib/virtuoso/db/virtuoso-temp.trx
MaxCheckpointRemap              = 2000
Striping                        = 0

;
;  Server parameters
;
[Parameters]
ServerPort                      = 1111
LiteMode                        = 0
DisableUnixSocket               = 1
DisableTcpSocket                = 0
;SSLServerPort                  = 2111
;SSLCertificate                 = cert.pem
;SSLPrivateKey                  = pk.pem
;X509ClientVerify               = 0
;X509ClientVerifyDepth          = 0
;X509ClientVerifyCAFile         = ca.pem
MaxClientConnections            = 10
CheckpointInterval              = 60
O_DIRECT                        = 0
CaseMode                        = 2
MaxStaticCursorRows             = 5000
CheckpointAuditTrail            = 0
AllowOSCalls                    = 0
SchedulerInterval               = 10
DirsAllowed                     = ., /app/virtuoso/share/virtuoso/vad, /usr/share/proj
ThreadCleanupInterval           = 0
ThreadThreshold                 = 10
ResourcesCleanupInterval        = 0
FreeTextBatchSize               = 100000
SingleCPU                       = 0
VADInstallDir                   = /app/virtuoso/share/virtuoso/vad/
PrefixResultNames               = 0
RdfFreeTextRulesSize            = 100
IndexTreeMaps                   = 256
MaxMemPoolSize                  = 200000000
PrefixResultNames               = 0
MacSpotlight                    = 0
IndexTreeMaps                   = 64
MaxQueryMem                     = 4G            ; memory allocated to query processor
VectorSize                      = 1000          ; initial parallel query vector (array of query operations) size
MaxVectorSize                   = 1000000       ; query vector size threshold.
AdjustVectorSize                = 0
ThreadsPerQuery                 = 4
AsyncQueueMaxThreads            = 10
;;
;; When running with large data sets, one should configure the Virtuoso
;; process to use between 2/3 to 3/5 of free system memory and to stripe
;; storage on all available disks.
;;
;; Uncomment next two lines if there is 2 GB system memory free
;NumberOfBuffers          = 170000
;MaxDirtyBuffers          = 130000
;; Uncomment next two lines if there is 4 GB system memory free
;NumberOfBuffers          = 340000
; MaxDirtyBuffers          = 250000
;; Uncomment next two lines if there is 8 GB system memory free
;NumberOfBuffers          = 680000
;MaxDirtyBuffers          = 500000
;; Uncomment next two lines if there is 16 GB system memory free
NumberOfBuffers          = 1360000
MaxDirtyBuffers          = 1000000
;; Uncomment next two lines if there is 32 GB system memory free
;NumberOfBuffers          = 2720000
;MaxDirtyBuffers          = 2000000
;; Uncomment next two lines if there is 48 GB system memory free
;NumberOfBuffers          = 4000000
;MaxDirtyBuffers          = 3000000
;; Uncomment next two lines if there is 64 GB system memory free
;NumberOfBuffers          = 5450000
;MaxDirtyBuffers          = 4000000
;;
;; Note the default settings will take very little memory
;; but will not result in very good performance
;;
;NumberOfBuffers          = 10000
;MaxDirtyBuffers          = 6000
MaxSortedTopRows         = 100000

[HTTPServer]
ServerPort                      = 80
ServerRoot                      = /app/virtuoso/var/lib/virtuoso/vsp
MaxClientConnections            = 10
DavRoot                         = DAV
EnabledDavVSP                   = 0
HTTPProxyEnabled                = 0
TempASPXDir                     = 0
DefaultMailServer               = localhost:25
ServerThreads                   = 10
MaxKeepAlives                   = 10
KeepAliveTimeout                = 10
MaxCachedProxyConnections       = 10
ProxyConnectionCacheTimeout     = 15
HTTPThreadSize                  = 280000
HttpPrintWarningsInOutput       = 0
Charset                         = UTF-8
;HTTPLogFile                    = logs/http.log
MaintenancePage                 = atomic.html
EnabledGzipContent              = 1

[AutoRepair]
BadParentLinks                  = 0

[Client]
SQL_PREFETCH_ROWS               = 100
SQL_PREFETCH_BYTES              = 16000
SQL_QUERY_TIMEOUT               = 0
SQL_TXN_TIMEOUT                 = 0
;SQL_NO_CHAR_C_ESCAPE           = 1
;SQL_UTF8_EXECS                 = 0
;SQL_NO_SYSTEM_TABLES           = 0
;SQL_BINARY_TIMESTAMP           = 1
;SQL_ENCRYPTION_ON_PASSWORD     = -1

[VDB]
ArrayOptimization               = 0
NumArrayParameters              = 10
VDBDisconnectTimeout            = 1000
KeepConnectionOnFixedThread     = 0

[Replication]
ServerName                      = db-AWSACRNVAL0003
ServerEnable                    = 1
QueueMax                        = 50000

;
;  Striping setup
;
;  These parameters have only effect when Striping is set to 1 in the
;  [Database] section, in which case the DatabaseFile parameter is ignored.
;
;  With striping, the database is spawned across multiple segments
;  where each segment can have multiple stripes.
;
;  Format of the lines below:
;    Segment<number> = <size>, <stripe file name> [, <stripe file name> .. ]
;
;  <number> must be ordered from 1 up.
;
;  The <size> is the total size of the segment which is equally divided
;  across all stripes forming  the segment. Its specification can be in
;  gigabytes (g), megabytes (m), kilobytes (k) or in database blocks
;  (b, the default)
;
;  Note that the segment size must be a multiple of the database page size
;  which is currently 8k. Also, the segment size must be divisible by the
;  number of stripe files forming  the segment.
;
;  The example below creates a 200 meg database striped on two segments
;  with two stripes of 50 meg and one of 100 meg.
;
;  You can always add more segments to the configuration, but once
;  added, do not change the setup.
;
[Striping]
Segment1                        = 100M, db-seg1-1.db, db-seg1-2.db
Segment2                        = 100M, db-seg2-1.db
;...

;[TempStriping]
;Segment1                       = 100M, db-seg1-1.db, db-seg1-2.db
;Segment2                       = 100M, db-seg2-1.db
;...

;[Ucms]
;UcmPath                        = <path>
;Ucm1                           = <file>
;Ucm2                           = <file>
;...

[Zero Config]
ServerName                      = virtuoso (AWSACRNVAL0003)
;ServerDSN                      = ZDSN
;SSLServerName                  =
;SSLServerDSN                   =

[Mono]
;MONO_TRACE                     = Off
;MONO_PATH                      = <path_here>
;MONO_ROOT                      = <path_here>
;MONO_CFG_DIR                   = <path_here>
;virtclr.dll                    =

[URIQA]
DynamicLocal                    = 0
DefaultHost                     = localhost:80

[SPARQL]
;ExternalQuerySource            = 1
;ExternalXsltSource             = 1
;DefaultGraph                   = http://localhost:8890/dataspace
;ImmutableGraphs                = http://localhost:8890/dataspace
ResultSetMaxRows                = 100000
MaxQueryCostEstimationTime      = 40000 ; in seconds
MaxQueryExecutionTime           = 60    ; in seconds
DefaultQuery                    = select distinct ?Concept where {[] a ?Concept} LIMIT 100
DeferInferenceRulesInit         = 0  ; controls inference rules loading
;PingService                    = http://rpc.pingthesemanticweb.com/

[Plugins]
;LoadPath                       = /app/virtuoso/lib/virtuoso/hosting
;Load1                          = plain, wikiv
Load2                           = plain, mediawiki
;Load3                          = plain, creolewiki
;Load4                  = plain, im
;Load5          = plain, wbxml2
;Load6                  = attach, libphp5.so
;Load7                  = Hosting, hosting_php.so
;Load8          = plain, proj4
;Load9                  = plain, geos
;Load10         = plain, shapefileio
;Load11                 = plain, hslookup
;Load12         = Hosting,hosting_perl.so
;Load13         = Hosting,hosting_python.so
;Load14         = Hosting,hosting_ruby.so
;Load15                         = msdtc,msdtc_sample
pkleef commented 2 years ago

One of the reasons for the HTTP status code 503 is the fact that Virtuoso from time to time needs to perform a checkpoint. As soon as virtuoso starts a checkpoint, the http server will return HTTP status code 503 on each request, until the checkpoint has completed.

Assuming your application fills the database by performing inserts/queries via the /sparql endpoint, this is a likely scenario.

You could check the virtuoso.log file to see if the times when you got 503 errors matches the checkpoint times in your log.

According to this Wikipedia article on HTTP status codes the 503 status means:

503 Service Unavailable
The server cannot handle the request (because it is overloaded or down for maintenance). 
Generally, this is a temporary state.

An HTTP aware application is supposed sleep for a couple of minutes before retrying the same operation until the operation succeeds.

If this only happens during the ingestion phase you can perform the following steps:

  1. start virtuoso
  2. use the isql (sometimes called isql-vt) to run the following statement:
SQL> checkpoint_interval(0);

  1. perform your operations and see if they now all succeed without getting a 503.

  2. once the operations are completed run the following commands in isql

SQL> checkpoint; 
SQL> checkpoint_interval(60);

If this resolves your problem, we can help adjusting the CheckpointInterval time in your virtuoso.ini or discuss other options.

plegault3397 commented 2 years ago

@pkleef When using Virtuoso Version: 07.20.3230, I did not experience this issue. Is this new to Virtuoso Version: 07.20.3233? Yes this is the initial ingestion of the data. is the interval in minutes? The ingestion phase for the current server takes over 8hr

pkleef commented 2 years ago

Checkpoint has been part of virtuoso since version 3 or 4 at least.

At this time i have no idea why you did not have this with the older v7 binary.

The interval is in minutes, but setting it to 0 like i showed you via the isql tool or setting it in the virtuoso.ini disables the automatic checkpointing.

Note this does not influence the integrity of the database, however it can cause delays in starting the database when it has not been properly shut down.

Let me know if the above solution solves your problem, else i will see how else we can figure out what is going on.

plegault3397 commented 2 years ago

Does not make sense that I use the older version of virtuoso and it runs 8+ hrs without interruption, when I switch the same server to the new version of virtuoso, and it stops with the error. It seems the pause from the MediaWiki server is happening in the old but not the new. Thanks for clarifying what the cause is.

plegault3397 commented 2 years ago

@pkleef: I downloaded the tar file Virtuoso Open Source Edition v7.2.6 and installed it fresh. Why does the version still show Virtuoso Version: 07.20.3233?

plegault3397 commented 2 years ago

I'm not sure what the issue is with the latest build from git hub,https://github.com/openlink/virtuoso-opensource.git.

However I downloaded the tar https://github.com/openlink/virtuoso-opensource/releases/download/v7.2.6.1/virtuoso-opensource-7.2.6.tar.gz instead and now it work as it should.