search

cfsimplicity / spreadsheet-cfml

Standalone library for working with spreadsheets and CSV in CFML
MIT License
125 stars 35 forks source link

First Time Install java.lang.NullPointerException #294

Open Conrad-T-Pino opened 2 years ago

Conrad-T-Pino commented 2 years ago

Any help interpreting this stack trace is welcome: "Severity","ThreadID","Date","Time","Application","Message" "ERROR","catalina-exec-3","07/05/2022","08:28:40","DelekHope","excel.cfm;java.lang.NullPointerException;lucee.runtime.exp.NativeException: java.lang.NullPointerException at lucee.runtime.op.Caster.castTo(Caster.java:4683) at lucee.runtime.functions.string.JavaCast.to(JavaCast.java:93) at lucee.runtime.functions.string.JavaCast.call(JavaCast.java:54) at helpers.sheet_cfc$cf.udfCall4(/helpers/sheet.cfc:338) at helpers.sheet_cfc$cf.udfCall(/helpers/sheet.cfc) at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106) at lucee.runtime.type.UDFImpl._call(UDFImpl.java:344) at lucee.runtime.type.UDFImpl.call(UDFImpl.java:217) at lucee.runtime.type.scope.UndefinedImpl.call(UndefinedImpl.java:785) at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:787) at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1747) at helpers.sheet_cfc$cf.udfCall2(/helpers/sheet.cfc:149) at helpers.sheet_cfc$cf.udfCall(/helpers/sheet.cfc) at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106) at lucee.runtime.type.UDFImpl._call(UDFImpl.java:344) at lucee.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:207) at lucee.runtime.type.scope.UndefinedImpl.callWithNamedValues(UndefinedImpl.java:802) at lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:866) at lucee.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:1766) at helpers.sheet_cfc$cf.udfCall3(/helpers/sheet.cfc:238) at helpers.sheet_cfc$cf.udfCall(/helpers/sheet.cfc) at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106) at lucee.runtime.type.UDFImpl._call(UDFImpl.java:344) at lucee.runtime.type.UDFImpl.call(UDFImpl.java:217) at lucee.runtime.ComponentImpl._call(ComponentImpl.java:650) at lucee.runtime.ComponentImpl._call(ComponentImpl.java:572) at lucee.runtime.ComponentImpl.call(ComponentImpl.java:1911) at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:787) at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1747) at spreadsheet_cfc$cf.udfCall9(/Spreadsheet.cfc:1416) at spreadsheet_cfc$cf.udfCall(/Spreadsheet.cfc) at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106) at lucee.runtime.type.UDFImpl._call(UDFImpl.java:344) at lucee.runtime.type.UDFImpl.call(UDFImpl.java:217) at lucee.runtime.type.scope.UndefinedImpl.call(UndefinedImpl.java:785) at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:787) at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1747) at spreadsheet_cfc$cf.udfCall7(/Spreadsheet.cfc:1234) at spreadsheet_cfc$cf.udfCall(/Spreadsheet.cfc) at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106) at lucee.runtime.type.UDFImpl._call(UDFImpl.java:344) at lucee.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:207) at lucee.runtime.ComponentImpl._call(ComponentImpl.java:651) at lucee.runtime.ComponentImpl._call(ComponentImpl.java:572) at lucee.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:1930) at lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:866) at lucee.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:1766) at webreg.delekhope.admin.action.excel_cfm$cf$4.call(/webreg/DelekHope/Admin/action/excel.cfm:17) at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1003) at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:926) at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:217) at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:44) at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2460) at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2450) at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2421) at lucee.runtime.engine.Request.exe(Request.java:45) at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1179) at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1125) at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97) at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51) at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:667) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:404) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:74) at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NullPointerException ... 82 more"

cfsimplicity commented 2 years ago

What does your calling code look like?

Conrad-T-Pino commented 2 years ago

local = [:]; local["Spreadsheet"] = new Spreadsheet(); local["SpreadsheetNew"] = local.Spreadsheet.new(sheetName="Sheet1");

Conrad-T-Pino commented 2 years ago

I've been back tracing the source; all these calls are inner methods of the download I really want;

local.Spreadsheet.workbookFromQuery(local.result); local.Spreadsheet.binaryFromQuery(local.result); local.Spreadsheet.downloadFileFromQuery(local.result, "FileName");

cfsimplicity commented 2 years ago

Not sure I understand why you're using the local scope in that way. local is a built in scope within a function and doesn't need to be defined. You also don't need to use [] notation when defining variables.

local.Spreadsheet = new Spreadsheet();

Or more simply, just use the var keyword once to declare it as local, then you can omit the scope:

var Spreadsheet = new Spreadsheet();
var newSpreadsheet = Spreadsheet.new(sheetName="Sheet 1");

In any case, which line of your code is actually triggering the exception?

Conrad-T-Pino commented 2 years ago

I'm in simple template "excel.cfm" and use "local" structure as fast way to clean up variables scope when including this template. I use "[]" notation to force name case in cfdump output.

All these cause same exception: local.Spreadsheet.new(sheetName="Sheet1"); local.Spreadsheet.workbookFromQuery(local.result); local.Spreadsheet.binaryFromQuery(local.result); local.Spreadsheet.downloadFileFromQuery(local.result, "FileName");

cfsimplicity commented 2 years ago

I've tried replicating your dummy "local scope" in a template and it doesn't seem to affect things so we can rule that out. But I can't replicate the NPE unfortunately which makes it tricky to diagnose.

Which versions of Lucee and java are you using?

Conrad-T-Pino commented 2 years ago

Lucee 5.3.8.206 openjdk version "11.0.15" 2022-04-19 OpenJDK Runtime Environment (build 11.0.15+10-post-Debian-1deb11u1) OpenJDK 64-Bit Server VM (build 11.0.15+10-post-Debian-1deb11u1, mixed mode, sharing)

cfsimplicity commented 2 years ago

Those versions should be fine. Sorry am at a bit of a loss...

Conrad-T-Pino commented 2 years ago

The stack trace contains "(/helpers/sheet.cfc:338)" as last location within this project's scope. The code at Line 338

336 private numeric function getSheetIndexFromName( required workbook, required string sheetName ){ 337 //returns -1 if non-existent 338 return arguments.workbook.getSheetIndex( JavaCast( "string", arguments.sheetName ) ); 339 }

The stack trace continue in "JavaCast" which suggests a "arguments.sheetName" issue?

cfsimplicity commented 2 years ago

Yes, but I've tested your code with the sheetName="Sheet1" argument and it works fine for me (and many other people are using the library without issue). It must be something in your environment, but it's hard to know what given that the NPE doesn't give any clue as to the underlying problem.

Presumably leaving off the sheetName argument makes no difference given that you said other calls are producing the same error?

cfsimplicity commented 2 years ago

Are you able to run the following? If so please post the output.

dump( new Spreadsheet().getEnvironment() )

Also, just a shot in the dark, but try this in case there's an OSGi issue of some kind:

new Spreadsheet().flushOsgiBundle()

Then retry your code.

Conrad-T-Pino commented 2 years ago

For expediency sake we are going to export "ms" styled CSS in html tables using excel "Content-Type". IMO less than ideal but good enough for this project's program manager.

My issue title choice is intended to imply I agree this issue is likely installation environment specific. I will do the heavy lifting following experienced guidance all hopefully to the project's and future user's benefit. If agreeable, I can deploy AWS clone in about an hour or two.

cfsimplicity commented 2 years ago

Did you try flushOsgiBundle() and getEnvironment()?

Conrad-T-Pino commented 2 years ago

No, I reverted local project host and can't replicate locally which is reason I propose the AWS Lucee clone.

cfsimplicity commented 2 years ago

Not sure I understand. So there's no issue on your local setup? Where is it a problem then? In AWS?

Conrad-T-Pino commented 2 years ago

I develop locally with VMware ESXi instance. Only try was local ESXi instance which I reverted. Integration testing is AWS EC2 instance. Production is another EC2 instance. What matters are AWS instances.

Conrad-T-Pino commented 2 years ago

BTW I see similar issue with GitHub "cfspreadsheet-lucee-5" Lucee extension which has Apache POI dependency as well.

cfsimplicity commented 2 years ago

Can you please provide a link to that issue?

Conrad-T-Pino commented 2 years ago

I didn't open cfspreadsheet-lucee-5 issue as I prefer this project's features.

The export HTML as Content-Type "application/msexcel" is good enough for browser and Excel but was labor intensive to mimic this project's "downloadFileFromQuery" function. I want to replace my hack with this project once this issue is resolved.

cfsimplicity commented 2 years ago

I just want to know which is the "similar issue" with the cfspreadsheet extension that you saw? Can you give the URL of that issue? Thanks.

jphustman commented 2 months ago

Hi, I'm running into this issue now -

just doing this:

spreadsheet = New spreadsheetLibrary.Spreadsheet();
workbook = spreadsheet.new();

The NullPointerException is here:

/var/www/app/spreadsheetLibrary/helpers/sheet.cfc: line 349
347:    private numeric function getSheetIndexFromName( required workbook, required string sheetName ){
348:       //returns -1 if non-existent
349:       **return arguments.workbook.getSheetIndex( JavaCast( "string", arguments.sheetName ) );**
350:    }
351:

I can write out like this and it's fine

        private numeric function getSheetIndexFromName( required workbook, required string sheetName ){
                //returns -1 if non-existent
                writeOutput(arguments.sheetName & "<br>");
                writeOutput(arguments.workbook.getSheetIndex(JavaCast( "string", arguments.sheetName)));
                abort;
                return arguments.workbook.getSheetIndex( JavaCast( "string", arguments.sheetName ) );
        }

Result:

Sheet1 (I can also initialize with new(sheetname="MySheet") and it will always return the name)
0

but when I put it back to the 'return' line that NullPointerException pops up

I tried the above recommendation flushOsgiBundle but it didn't do anything.

Here is my getEnvironment():

DATEFORMATS Date: string: yyyy-mm-dd DateTime: string: yyyy-mm-dd HH:nn:ss Time: string: hh:mm:ss Timestamp: string: yyyy-mm-dd hh:mm:ss |

ENGINE | string Lucee 6.0.3.1

JAVACLASSESLASTLOADEDVIA | string Nothing loaded yet

JAVALOADERDOTPATH | string javaLoader.JavaLoader

JAVALOADERNAME | string

OSGILIBBUNDLEVERSION | string 5.2.5.1

POIVERSION | string 5.2.5

REQUIRESJAVALOADER | boolean false

VERSION | string 4.0.0

/etc/lsb-release: DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"

java -version: openjdk version "11.0.22" 2024-01-16 OpenJDK Runtime Environment (build 11.0.22+7-post-Ubuntu-0ubuntu220.04.1) OpenJDK 64-Bit Server VM (build 11.0.22+7-post-Ubuntu-0ubuntu220.04.1, mixed mode, sharing)

But what is also interesting is I have very similar servers running Lucee and it runs fine so it's something specific - all settings are identical, I am just not sure how to troubleshoot beyond this point, since outputting the line that is being returned shows up correctly, "returning" throws that error.

Here is the full 'java' stacktrace:

lucee.runtime.exp.NativeException: java.lang.NullPointerException
  at lucee.runtime.op.Caster.castTo(Caster.java:5025)
  at lucee.runtime.functions.string.JavaCast.to(JavaCast.java:93)
  at lucee.runtime.functions.string.JavaCast.call(JavaCast.java:54)
  at procurement.awp.spreadsheetlibrary.helpers.sheet_cfc$cf.udfCall4(/app/spreadsheetLibrary/helpers/sheet.cfc:349)
  at procurement.awp.spreadsheetlibrary.helpers.sheet_cfc$cf.udfCall(/app/spreadsheetLibrary/helpers/sheet.cfc)
  at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
  at lucee.runtime.type.UDFImpl._call(UDFImpl.java:356)
  at lucee.runtime.type.UDFImpl.call(UDFImpl.java:223)
  at lucee.runtime.type.scope.UndefinedImpl.call(UndefinedImpl.java:787)
  at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:797)
  at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1999)
  at procurement.awp.spreadsheetlibrary.helpers.sheet_cfc$cf.udfCall2(/app/spreadsheetLibrary/helpers/sheet.cfc:151)
  at procurement.awp.spreadsheetlibrary.helpers.sheet_cfc$cf.udfCall(/app/spreadsheetLibrary/helpers/sheet.cfc)
  at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
  at lucee.runtime.type.UDFImpl._call(UDFImpl.java:356)
  at lucee.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:213)
  at lucee.runtime.type.scope.UndefinedImpl.callWithNamedValues(UndefinedImpl.java:805)
  at lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:876)
  at lucee.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:2018)
  at procurement.awp.spreadsheetlibrary.helpers.sheet_cfc$cf.udfCall3(/app/spreadsheetLibrary/helpers/sheet.cfc:237)
  at procurement.awp.spreadsheetlibrary.helpers.sheet_cfc$cf.udfCall(/app/spreadsheetLibrary/helpers/sheet.cfc)
  at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
  at lucee.runtime.type.UDFImpl._call(UDFImpl.java:356)
  at lucee.runtime.type.UDFImpl.call(UDFImpl.java:223)
  at lucee.runtime.ComponentImpl._call(ComponentImpl.java:668)
  at lucee.runtime.ComponentImpl._call(ComponentImpl.java:590)
  at lucee.runtime.ComponentImpl.call(ComponentImpl.java:2014)
  at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:797)
  at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1999)
  at procurement.awp.spreadsheetlibrary.spreadsheet_cfc$cf.udfCalla(/app/spreadsheetLibrary/Spreadsheet.cfc:1315)
  at procurement.awp.spreadsheetlibrary.spreadsheet_cfc$cf.udfCall(/app/spreadsheetLibrary/Spreadsheet.cfc)
  at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
  at lucee.runtime.type.UDFImpl._call(UDFImpl.java:356)
  at lucee.runtime.type.UDFImpl.call(UDFImpl.java:223)
  at lucee.runtime.type.scope.UndefinedImpl.call(UndefinedImpl.java:787)
  at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:797)
  at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1999)
  at procurement.awp.spreadsheetlibrary.spreadsheet_cfc$cf.udfCall8(/app/spreadsheetLibrary/Spreadsheet.cfc:1077)
  at procurement.awp.spreadsheetlibrary.spreadsheet_cfc$cf.udfCall(/app/spreadsheetLibrary/Spreadsheet.cfc)
  at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
  at lucee.runtime.type.UDFImpl._call(UDFImpl.java:356)
  at lucee.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:213)
  at lucee.runtime.type.scope.UndefinedImpl.callWithNamedValues(UndefinedImpl.java:805)
  at lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:876)
  at lucee.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:2018)
  at procurement.awp.spreadsheetlibrary.spreadsheet_cfc$cf.udfCall8(/app/spreadsheetLibrary/Spreadsheet.cfc:1103)
  at procurement.awp.spreadsheetlibrary.spreadsheet_cfc$cf.udfCall(/app/spreadsheetLibrary/Spreadsheet.cfc)
  at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
  at lucee.runtime.type.UDFImpl._call(UDFImpl.java:356)
  at lucee.runtime.type.UDFImpl.call(UDFImpl.java:223)
  at lucee.runtime.ComponentImpl._call(ComponentImpl.java:668)
  at lucee.runtime.ComponentImpl._call(ComponentImpl.java:590)
  at lucee.runtime.ComponentImpl.call(ComponentImpl.java:2014)
  at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:797)
  at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1999)
  at procurement.awp.awp_tas_xls_cfm540$cf.call(/app/awp-tas-xls.cfm:96)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1029)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:952)
  at lucee.runtime.listener.ClassicAppListener._onRequest(ClassicAppListener.java:65)
  at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:45)
  at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2716)
  at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2702)
  at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2673)
  at lucee.runtime.engine.Request.exe(Request.java:45)
  at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1259)
  at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1205)
  at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)
  at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
  at mod_cfml.core.invoke(core.java:180)
  at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:388)
  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
  at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
  at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
  at java.base/java.lang.Thread.run(Thread.java:829)
 Caused by: java.lang.NullPointerException
  ... 92 more
cfsimplicity commented 2 months ago

Thanks for the detailed report. Unfortunately I'm still at a complete loss. As with the OP I guess it's a specific issue of some kind with your environment. Is the troublesome instance in AWS, like the OP?

jphustman commented 2 months ago

Thank you. I'll keep troubleshooting and explain what is going on here when I figure it out (I will figure it out). These are virtual machines in Azure, and they are 'base' Ubuntu images that we have set up - I get handed a fresh installation and then do what I need to with them. What is interesting (as I said previously) is that other servers I have running the same operating systems with Lucee (very similar as in dev/test/prod) work fine, so it is something specific with something on this server. These are also 'backend' servers with Nginx Plus sitting in front. I assume there is something with the Java (though they are all running the same version, installed from the Ubuntu repositories) so I'll keep digging. So it's not your code (and thank you for writing this by the way!)

cfsimplicity commented 2 months ago

OK, thanks. Do post back if you come up with anything. As you say it's likely to be a specific java/JVM config issue of some kind.

jphustman commented 2 months ago

Ok, I figured out my problem, though I can't say it was the original poster's problem; maybe if someone else comes across this, we can verify. I even went and downloaded the Lucee codebase locally and added debugging statements deep into the 'Caster' code, and the string variable was also simply just disappearing there, which was leading me to believe it was even lower level than that - so I downloaded the OpenJDK-11 code from the source website (as opposed to the JDK installed from the Ubuntu repositories) and pointed Lucee to that, but the problem still persisted. Then I was like "ok this has to be simple let's go back to the basics" - in the parent directory I looked at the Application.cfc file (which I probably should have done in the beginning) and there was this:

<cfapplication
    name="My-Application"
    sessionmanagement="Yes"
    sessiontimeout="#CreateTimeSpan(0,0,20,0)#"
    --> nullSupport="true" <--
>

I removed nullSupport="true" and the problem disappeared, and the spreadsheet was created normally.

I am enabling Full Null Support in the Lucee Administrator, as it is recommended, along with the other recommendations, and then searching our application code base for that same attribute (and enableNullSupport) and removing it.

I learned quite a bit about the Lucee codebase in this journey.

Edit: Actually, after some more testing enabling Full Null Support is what breaks it.

cfsimplicity commented 2 months ago

Interesting. Issues with null support have come up before.

I've just run the test suite on a commandbox engine with null support enabled though and I'm not seeing any failures.

You mentioned that only one of several similar servers had the issue. Do the others have null support enabled?

jphustman commented 2 months ago

Yeah, I can't get it to fail on purpose. Let me do a little more investigating and try to figure out a test case...

jphustman commented 2 months ago

Ok I figured out our problem, there is a null value in a sql query response, try this:

It throws NPE here:

The Error Occurred in
\app\spreadsheetLibrary\helpers\cell.cfc: line 213
211:
212:    private any function setStringValue( required any cell, required any value ){
213:       arguments.cell.setCellValue( JavaCast( "string", arguments.value ) );
214:       return this;
215:    }
<cfscript>
    // Dummy query
    news = queryNew("id,title", "integer,varchar");

    // Dummy data
    queryAddRow(news);
    querySetCell(news, "id", 1);
    querySetCell(news, "title", "Dewey defeats Truman");

    queryAddRow(news);
    querySetCell(news, "id", 2);
    querySetCell(news, "title", null);

    queryAddRow(news);
    querySetCell(news, "id", 3);
    querySetCell(news, "title", "Men walk on Moon");

    // QofQ
    sortedNews = queryExecute("
        SELECT id, title
        FROM news
        ORDER BY title
        DESC
    ", [], {dbtype="query"});

    // Create a spreadsheet and add rows
    spreadsheet = new spreadsheetLibrary.Spreadsheet();
    workbook = spreadsheet.new();
    spreadsheet.addRows(workbook, sortedNews);

    filename = "Example.xls";
    spreadsheet.download(workbook, filename);
</cfscript>
cfsimplicity commented 2 months ago

Ok I figured out our problem, there is a null value in a sql query response

Good work.

Unfortunately it turns out I was completely wrong about the tests passing with null support. Actually the setting was not enabled as I thought. I've tried them again with it definitely enabled and they completely bomb. There are a large number of areas of code which just don't work with null support on.

In the past when the project was smaller I managed to work through those areas to make them null-support compatible, but I'm afraid there are just too many now. I've tried tackling them but keep running into obscure NPEs which I can't fathom.

My only suggestion for now is to turn off null support if you possibly can. You say that it's "recommended" but frankly I doubt if many people use it for this very reason: it breaks otherwise perfectly working code.

JamoCA commented 2 months ago

Explicit NULL handling requires a lot of extra background work as it first requires testing to determine if the variable exists and then following up with an isNull() check. Normally keyExists & isDefined will return false and isNull will return true. The key of a NULL value is returned when using structKeyList and can cause errors if you blindly iterate over without additional per-key testing.

I try to write cross-compatible CFML. Adobe has added a number of system & application flags, changed default algorithms, etc... there's no efficient way to check flags to determine how to work around a non-default behavior that may not have been taken into consideration. What happens when a third-party library is dependent on "default" options, but an environment has some flags set to modify the default behaviors? #chaos