[Feature request]: JASP should support more than 1000 columns

JeffS1984 commented 11 months ago

JASP Version

0.18.0

Commit ID

No response

JASP Module

Unrelated

What analysis are you seeing the problem on?

No response

What OS are you seeing the problem on?

Windows 11

Bug Description

I have been using JASP for a couple years to teach undergrad stats. This is the first semester I've had student issues who are using Windows. I've had a few who are unable to get Jasp to upload a .csv data set. The error they receive refers to "alter tables". I have attempted to work with these students to get Jasp 18 totally deleted from their computers and download a slightly older version like 17.2. However, their computers keep saying they already have Jasp downloaded, even though we cannot find Jasp anywhere on there.

Expected Behaviour

The .csv file should pull up on Jasp

Steps to Reproduce

Download Jasp 18 on Windows
Attempt to upload a .csv file
I'm not sure if this occurs for all Windows users -- I've only heard of an issue from 2-3 students out of 85-90. ...

Log (if any)

No response

Final Checklist

[ ] I have included a screenshot showcasing the issue, if possible.
[ ] I have included a JASP file (zipped) or data file that causes the crash/bug, if applicable.
[X] I have accurately described the bug, and steps to reproduce it.

boutinb commented 11 months ago

Hi @JeffS1984 This is quite annoying, but to be able to help you, we need a reproduction path. Without it, we cannot reproduce this error, and it is quite hard to see what is going wrong.

JeffS1984 commented 10 months ago

My student was able to get past the issue. However, I updated my JASP to version 18 and am now running into the same thing. I have a Mac -- Ventura 13.3.3.1 and Apple M1 chip.

I am attaching a picture of the error when trying to upload a .csv file that has worked just fine with previous versions of Jasp.

I also noticed that on a data set that Jasp would upload, it was not producing basic output for me. I did a screenshot of that as well.

Thanks. 247BDE83-F18D-424C-A937-D4AB10F48B99 05222013-2D0A-48FD-AF91-58A2DB6284A0

boutinb commented 10 months ago

Hi @JeffS1984 How many columns have your datafile? The error seems to indicate that there are more than 1000 columns. We know about this limitation, and we want to add some setting to circumvent this problem, but first just to be sure: have you more than 1000 columns. If yes, would it be possible to split this datafile with less columns?

ertugruluyar-github commented 9 months ago

I got same issue when I tried to open PISA 2022 student data. I am using Linux and I installed JASP via flatpak (flathub).

boutinb commented 9 months ago

Hi @ertugruluyar-github, does have this PISA 2022 student data more than 1000 columns?

ertugruluyar-github commented 8 months ago

Hi @ertugruluyar-github, does have this PISA 2022 student data more than 1000 columns?

Hi @boutinb Yes, there is 1279 columns. I tried with Jamovi 2.4.11 also but the progress bar not filling and it doesn't give any error message and I wait hours and hours but it didn't open. I tried with GNU PSPP also and it did load the variables, but not the data. So, because of all softwares that I tried didn't able to open the .sav file, I write some python code with chatGPT and filtered data with country id. And Jamovi be able to open file (I haven't filtered the columns.). Some columns in the dataset not so needed, so I can write some code with chatGPT to filter only variables that ı specified. So that, JASP could open the dataset. But why 1000 columns limit is there?

To be clear, I want to open the file because I wanted the filter datates with country id then cut out the the dataset to have small .sav file to analyse it. I want to have a small .sav beacuse my laotop not so powerful and 1.4 GB PISA 2015 file opened with Jamovi in like 30-40 minutes.

To be noted, I wanna write my thesis with Hiyerarcihal Linear Modelling (Multilevel Modelling) and in JASP there is a tool for that but in this tool I didn't able to analyse ANOVA model (fully unconditional model). If it has this (Maybe I don't know to do this.), because it have the Open Science Framework (https://osf.io/) integration, I wanted to use it for my analysis even the Jamovi's multilevel anlysis tool a bit better for me.

tomtomme commented 7 months ago

@ertugruluyar-github @JeffS1984 just a status update. This is still valid with current 0.19 beta. It will not be forgotten.

JorisGoosen commented 4 months ago

Some columns in the dataset not so needed, so I can write some code with chatGPT to filter only variables that ı specified. So that, JASP could open the dataset. But why 1000 columns limit is there?

Because we switched to using sqlite as an internal store and they apparently have a hardcoded maximum amount of columns in a dataset. We only noticed this after all the code was already written. Because usually people dont use JASP for such datasets. More things like R etcetera.

Im going to assume that even if it would load there might be other bottlenecks in the application (when loading it in R or something, not sure though).

We are working on syntaxmode, which might obviate the need for the sqlite storage and thus might be able to handle this many columns. Of course, supporting an arbitrary amount of columns is desirable, but given our aim at teaching and making statistics available for non-computer-expert-users we are focussing first on things like supporting data editing, speed and handling different kinds of data. The >1000 columns use-case is certainly a real one but it is a bit more niche so it might be a bit of time before we get to this

tomtomme commented 4 months ago

@JeffS1984 Can you alter the title of this issue to reflect that it is now a feature request to support > 1000 columns? Thx

JeffS1984 commented 4 months ago

Thanks for all the comments on here. I do teach undergraduates with Jasp, but I like to use large social surveys like the General Social Survey in the US, which is why this issue came up for me.

tomtomme commented 3 months ago

This issue is tracked via an internal issue here: https://github.com/jasp-stats/INTERNAL-jasp/issues/2421 It may be completed for version 0.20 or later. I will mark this issue as duplicate but keep it open for reference.

stvnbird commented 2 months ago

A previous post indicates this is a niche issue. Just wanted to mention that data sets with more than 1000 columns is very common in the social sciences (sociology, criminology, political science, etc.)--basically any field that uses large scale surveying. Perhaps that is not an intended target audience for JASP, though. Thank you for the great work you do here!

tomtomme commented 1 week ago

This should be fixed with https://github.com/jasp-stats/jasp-desktop/pull/5636 It will be hopefully released in some weeks with version 0.19.1 at least 16k columns should load fine then

JorisGoosen commented 1 week ago

0.19.1 is going to be the hotfix right? This is not in there.

@boutinb @RensDofferhoff

JeffS1984 commented 1 week ago

Still not fixed with 19 -- version 17.3 works for my purposes but hoping the next version has it fixed.

On Fri, Sep 6, 2024 at 11:16 AM Joris Goosen @.***> wrote:

0.19.1 is going to be the hotfix right? This is not in there.

@boutinb https://github.com/boutinb @RensDofferhoff https://github.com/RensDofferhoff

— Reply to this email directly, view it on GitHub https://github.com/jasp-stats/jasp-issues/issues/2327#issuecomment-2334281563, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVQYHV2BWRSZIBGLXZDXVD3ZVHBLPAVCNFSM6AAAAAA5U463SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZUGI4DCNJWGM . You are receiving this because you were mentioned.Message ID: @.***>

RensDofferhoff commented 5 days ago

@JeffS1984 You can download one of our nightlies marked 'development'. These have the fixes applied https://static.jasp-stats.org/Nightlies/

@JorisGoosen We could include it? it does not interfere with Rpkg state?

jasp-stats / jasp-issues