search

USACE / cwms-data-api

Corps Water Management System RESTful Data Service
MIT License
13 stars 14 forks source link

Unable to store rating curve with a single rating #423

Closed adamkorynta closed 1 year ago

adamkorynta commented 1 year ago

Error storing a single rating curve to the database:

14-Jul-2023 03:08:54.869 SEVERE [https-jsse-nio-8443-exec-589] cwms.cda.api.RatingController.create -5127655873716502772: Failed to process create request
    hec.data.RatingException: Text is not a valid compressed or uncompressed CWMS Ratings XML instance.
        at mil.army.usace.hec.cwms.rating.io.xml.RatingXmlFactory.ratingSet(RatingXmlFactory.java:294)
        at cwms.cda.api.RatingController.deserializeFromXml(RatingController.java:164)
        at cwms.cda.api.RatingController.deserializeRatingSet(RatingController.java:152)
        at cwms.cda.api.RatingController.deserializeRatingSet(RatingController.java:144)
        at cwms.cda.api.RatingController.deserializeRatingSet(RatingController.java:138)
        at cwms.cda.api.RatingController.create(RatingController.java:111)
        at io.javalin.apibuilder.CrudFunction$3.invoke$lambda-0(CrudHandler.kt:32)
        at io.javalin.apibuilder.CrudFunctionHandler.handle(CrudHandler.kt)
        at cwms.cda.security.KeyAccessManager.manage(KeyAccessManager.java:36)
        at cwms.cda.security.MultipleAccessManager.manage(MultipleAccessManager.java:41)
        at io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96)
        at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
        at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
        at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:99)
        at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
        at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
        at io.javalin.http.JavalinServletHandler.executeNextTask$lambda-11$lambda-10(JavalinServletHandler.kt:119)
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
        at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628)
        at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996)
        at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:119)
        at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
        at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
        at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
        at cwms.cda.ApiServlet.service(ApiServlet.java:510)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:357)
        at org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:176)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:660)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
        at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:261)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:750)
        Suppressed: java.lang.IllegalArgumentException: Length of input is not multiple of 4
            at hec.util.Base64.decode(Base64.java:122)
            at hec.util.Base64.decode(Base64.java:168)
            at hec.util.Base64.decode(Base64.java:59)
            at hec.util.Base64.decode(Base64.java:49)
            at hec.util.TextUtil.uncompress(TextUtil.java:422)
            at mil.army.usace.hec.cwms.rating.io.xml.RatingXmlFactory.ratingSet(RatingXmlFactory.java:289)
            ... 57 more
    Caused by: hec.data.RatingException: java.lang.NullPointerException
        at mil.army.usace.hec.cwms.rating.io.xml.RatingSetContainerXmlFactory.parseInputSource(RatingSetContainerXmlFactory.java:346)
        at mil.army.usace.hec.cwms.rating.io.xml.RatingSetContainerXmlFactory.parseReader(RatingSetContainerXmlFactory.java:316)
        at mil.army.usace.hec.cwms.rating.io.xml.RatingSetContainerXmlFactory.ratingSetContainerFromXml(RatingSetContainerXmlFactory.java:247)
        at mil.army.usace.hec.cwms.rating.io.xml.RatingSetContainerXmlFactory.ratingSetContainerFromXml(RatingSetContainerXmlFactory.java:237)
        at mil.army.usace.hec.cwms.rating.io.xml.RatingXmlFactory.ratingSet(RatingXmlFactory.java:285)
        ... 57 more
    Caused by: java.lang.NullPointerException
        at mil.army.usace.hec.cwms.rating.io.xml.RatingSetContainerXmlFactory.endElement(RatingSetContainerXmlFactory.java:1465)
        at org.xml.sax.helpers.ParserAdapter.endElement(ParserAdapter.java:652)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:602)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1784)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2969)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:113)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:507)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:867)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:796)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:142)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216)
        at org.xml.sax.helpers.ParserAdapter.parse(ParserAdapter.java:431)
        at org.xml.sax.helpers.XMLFilterImpl.parse(XMLFilterImpl.java:357)
        at mil.army.usace.hec.cwms.rating.io.xml.RatingSetContainerXmlFactory.parseInputSource(RatingSetContainerXmlFactory.java:338)
        ... 61 more

The rating xml sent to CDA is the following:

<?xml version="1.0" encoding="utf-8"?>
<ratings
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.hec.usace.army.mil/xmlSchema/cwms/Ratings.xsd">
    <simple-rating office-id="SWT">
        <rating-spec-id>07179500.Stage;Flow.EXSA.PRODUCTION</rating-spec-id>
        <units-id>ft;cfs</units-id>
        <effective-date>2023-07-14T03:10:00Z</effective-date>
        <transition-start-date/>
        <create-date>2023-07-14T03:10:00Z</create-date>
        <active>true</active>
        <description>NEOSHO R AT COUNCIL GROVE, KS Expanded, Shift-Adjusted PRODUCTION Stream Rating</description>
        <rating-points>
            <point>
                <ind>0.0</ind>
                <dep>0.0</dep>
            </point>
            <point>
                <ind>1.0</ind>
                <dep>2.0</dep>
            </point>
        </rating-points>
    </simple-rating>
</ratings>
MikeNeilson commented 1 year ago

@perrymanmd Thoughts? Glancing at it it looks like something is expecting some base64 encoding within the rating factory.

at hec.util.Base64.decode(Base64.java:49)
at hec.util.TextUtil.uncompress(TextUtil.java:422)
at mil.army.usace.hec.cwms.rating.io.xml.RatingXmlFactory.ratingSet(RatingXmlFactory.java:289)
... 57 more
adamkorynta commented 1 year ago

The XML parsing tries to parse as uncompressed first, but then if that fails it tries to uncompress and then parse. The uncompress exception is the suppressed one, while the original gets reported as the direct cause.

The issue here is the the ratings XML we are storing doesn't include the rating spec or template information so a full RatingSet object cannot be created. Since we are parsing the XML just to store it back to the db as XML I figured we might as well skip the step.

MikeNeilson commented 1 year ago

Ah okay, so it's doing the bad "um here let me try this other thing for you." Which you've just changed.