Stata running "jl" directly crashes

[ForbearChen]

I excitedly tried the setup, but encountered some issues after installation. Here are the details:

Platform: OS: Windows 11 Pro 64-bit RAM: 64.0GB CPU: 11th Gen Intel Core i7-11800H GPU: NVIDIA GeForce RTX 3070 Laptop GPU

The problem I encountered is that, after installing Julia 1.9.4 and the necessary packages (Vcov, FixedEffectModels, DataFrames, and CUDA), I followed the instructions from https://www.statalist.org/forums/forum/general-stata-discussion/general/1735303-reghdfe-10x-faster to install reghdfejl and using the command

• net install reghdfejl, replace from(https://raw.github.com/droodman/reghdfejl/v0.3.0)
• net install julia, replace from(https://raw.github.com/droodman/julia.ado/v0.5.8).

However, in both Stata 17.0 and Stata 18.0 MP versions, executing jl causes Stata to crash. The reghdfe command works fine, but reghdfejl results in a crash. Where should I start to resolve this issue? Alternatively, what additional information should I provide to supplement my problem?

Thank you.

[ForbearChen]

Additionally, it's worth noting that I installed Julia as instructed, using the compressed package downloaded from the official website, and it has been added to the PATH.

[droodman]

Thank you for the detailed feedback. Sorry to hear that!

I've been revamping how the Stata program searches for the needed Julia library file, "libjulia.dll". The new method should be more robust to variety in Julia installation. It is working for me on Windows and Linux. Let's see if it helps.

Please reinstall the julia package in Stata using the net install command in your message, but change the 0.5.8 to 0.6.0. I've marked 0.6.0 as a "pre-release."

Please confirm that you can run Julia from the command line, for example, by hitting the Windows logo button, typing cmd, to start the cmd shell, and then typing julia.

Then restart Stata and see if reghdfejl works.

If that still crashes. Try something simple like

jl: 1+1

in Stata. I predict that would crash too, but it would be good to confirm as an initial step in diagnosis.

[ForbearChen]

Thank you for the detailed feedback. Sorry to hear that!

I've been revamping how the Stata program searches for the needed Julia library file, "libjulia.dll". The new method should be more robust to variety in Julia installation. It is working for me on Windows and Linux. Let's see if it helps.

Please reinstall the julia package in Stata using the net install command in your message, but change the 0.5.8 to 0.6.0. I've marked 0.6.0 as a "pre-release."

Please confirm that you can run Julia from the command line, for example, by hitting the Windows logo button, typing cmd, to start the cmd shell, and then typing julia.

Then restart Stata and see if reghdfejl works.

If that still crashes. Try something simple like

jl: 1+1

in Stata. I predict that would crash too, but it would be good to confirm as an initial step in diagnosis.

Many thx for your reply. I installed and tried version 0.6.0, but it still causes Stata to crash. However, compared to the previous version, this time, as Stata window disappears, I can briefly see CMD flash by.

Furthermore, I've confirmed that I can run Julia from the command line. Additionally, I checked the installed packages using the command import Pkg; Pkg.installed(), and I can see that the required packages are installed, but they are not functioning correctly.

I look forward to your further assistance. If there are any specific tests you'd like me to perform, please feel free to contact me anytime.

[droodman]

Do you get more than one line when you type "where julia.exe" in cmd?

[droodman]

Also, is this computer your own, or is it managed by an organization, perhaps with tight security settings?

[droodman]

This is a tough one. @ForbearChen would you open to letting me take remote control of your computer some time so that I could try to debug? We could use the Quick Assist. The challenge is that the crash is probably happening in the C code, and the only way I know to debug it is to insert print and break statements in the C code, recompile, and test. If this is a possibility, it would be good for you to install Visual Studio Community 2022. You can contact me at the email address in the help file.

[droodman]

@ForbearChen, version 0.6.1 has a small change that might help, though I am not optimistic. Try installing 0.6.1 and restarting Stata.

[ForbearChen]

Thanks for your dedication and valuable contributions. I have now even tried version 0.6.1, but it still crashes after entering jl. I am using my own computer, and I will make further attempts. If necessary, I will email you to explore the possibility of remote control to see if we can find a solution that benefits more users. I hope your efforts will bring benefits to a wider audience.

[ForbearChen]

Do you get more than one line when you type "where julia.exe" in cmd?

I hadn't noticed this before. I just tried it, and it only provided one line of feedback, which is the installation path for my Julia: C:\Users\ForbearChen\AppData\Local\Programs\Julia-1.9.4\bin\julia.exe.

[ForbearChen]

I am doing my best to identify the root cause of the issue. I have noticed similar feedback from others on the forum, and it appears that a common factor is that our ado file paths are not in the default locations. Therefore, I am considering whether the issue might be related to not accounting for non-default paths. For instance, my current path is as follows. If this is the problem, addressing it could potentially assist in debugging for future versions.

. sysdir
   STATA:  C:\Program Files\Stata17\
    BASE:  C:\Program Files\Stata17\ado\base\
    SITE:  C:\Program Files\Stata17\ado\site\
    PLUS:  c:\ado\plus\
PERSONAL:  c:\ado\personal\
OLDPLACE:  c:\ado\

[droodman]

Thank you.

A one line response to “where julia” is good.

Non-standard Stata directories shouldn’t cause a crash. If Stata is crashing, that means Stata is finding jl.ado, running it, and then finding and loading the plugin file, “jl.plugin”. It’s after that that the trouble starts…

However, it would be good to make sure you don’t have more than one copy of jl.plugin floating around, to make sure it can only run the newest one.

What forum are you referring to?

--David

From: ForbearChen @.> Sent: Sunday, December 3, 2023 11:04 AM To: droodman/julia.ado @.> Cc: droodman @.>; Comment @.> Subject: Re: [droodman/julia.ado] Stata running "jl" directly crashes (Issue #2)

I am doing my best to identify the root cause of the issue. I have noticed similar feedback from others on the forum, and it appears that a common factor is that our ado file paths are not in the default locations. Therefore, I am considering whether the issue might be related to not accounting for non-default paths. For instance, my current path is as follows. If this is the problem, addressing it could potentially assist in debugging for future versions.

. sysdir STATA: C:\Program Files\Stata17\ BASE: C:\Program Files\Stata17\ado\base\ SITE: C:\Program Files\Stata17\ado\site\ PLUS: c:\ado\plus\ PERSONAL: c:\ado\personal\ OLDPLACE: c:\ado\

— Reply to this email directly, view it on GitHub https://github.com/droodman/julia.ado/issues/2#issuecomment-1837525867 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AGB2Z2NO5O62EBOT25DBN4LYHSPG3AVCNFSM6AAAAABAD3XJBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZXGUZDKOBWG4 . You are receiving this because you commented. https://github.com/notifications/beacon/AGB2Z2MEWUDDWWG3AZYK4DLYHSPG3A5CNFSM6AAAAABAD3XJBKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTNQZVWW.gif Message ID: @. @.> >

[ForbearChen]

Hi David, I checked the subfolder named 'j' within my ado directory, and there is only one file named 'jl.plugin'. Additionally, when I referred to the forum, I meant the post of #19 in the provided link(https://www.statalist.org/forums/forum/general-stata-discussion/general/1735303-reghdfe-10x-faster/page2).

[droodman]

Yes, there should just be a “jl.ado” and a “jl.plugin” file there. It’s possible that there are files with those names in some other folder too, which could create a problem. But I doubt it.

[droodman]

Make sure another package with a plugin works.

You can do

net sj 17-4 st0500
net install st0500
net get st0500
use smoking, clear
tsset state year
synth cigsale beer(1984(1)1988) lnincome retprice age15to24 cigsale(1988) cigsale(1980) cigsale(1975), trunit(3) trperiod(1989)

[ForbearChen]

I examined all the files under folders with names starting with other letters, and I did not find any files with the same names as these two. Regarding the st0500 you mentioned for synthetic control, it worked.

[droodman]

Does it crash if you start Stata and do

sysuse auto
jl PutVarsToDF

[ForbearChen]

Yes, it crashes. The issue occurs whenever commands related to "jl" are entered.

[droodman]

In Stata, type which jl. Then run or do the displayed file. For example, on my machine I type:

run"c:\ado\plus\j\jl.ado"

Then try

plugin call _julia

[ForbearChen]

No response at all, lol:)

[droodman]

Well, that means it's loading the plugin and calling it, which in itself doesn't cause a crash, so that good.

Now try:

jl start

The question is whether that crashes it or just gives no response. Again no response would be good.

[ForbearChen]

jl start

it crashed in this step

[droodman]

Great. I have hacked the plugin for debugging. Please reinstall with

net install julia, replace all from(https://raw.github.com/droodman/julia.ado/master)

Then restart Stata and try

jl start

Tell me if it crashes or just gives an error.

[droodman]

(You can see I inserted a return statement in the plugin code here to make it stop early. I'm trying to pinpoint where the crash happens.)

[ForbearChen]

Thank you for the timely debugging. After the update, at least, it no longer crashes. The outcome now looks like this. Best wishes!

[droodman]

Good. That means Stata is loading the plugin and the plugin is loading Julia.

Can you do exactly the same thing again? I modified the plugin again, so reinstall and test.

[droodman]

Also. Please load Julia directly and try this line. I am wondering if it gives an error message:

const _Stata_io = IOBuffer(); const _Stata_context=IOContext(_Stata_io, :limit=>true)

[ForbearChen]

Good. That means Stata is loading the plugin and the plugin is loading Julia.

Can you do exactly the same thing again? I modified the plugin again, so reinstall and test.

This time it will crash.

[ForbearChen]

Also. Please load Julia directly and try this line. I am wondering if it gives an error message:

const _Stata_io = IOBuffer(); const _Stata_context=IOContext(_Stata_io, :limit=>true)

julia> const _Stata_io = IOBuffer(); const _Stata_context=IOContext(_Stata_io, :limit=>true) IOContext(IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)) ![Uploading 3.png…]()

[ForbearChen]

[droodman]

Wow. So it is loading the Julia library, but failing to initialize it.

Do the same thing again. It should print out the path on your computer for libjulia.dll and then print an error message. Send me the path.

For example, on my computer i get

C:\Users\drood\.julia\juliaup\julia-1.9.4+0.x64.w64.mingw32\bin\libjulia.dll

[ForbearChen]

Wow. So it is loading the Julia library, but failing to initialize it.

Do the same thing again. It should print out the path on your computer for libjulia.dll and then print an error message. Send me the path.

For example, on my computer i get

C:\Users\drood\.julia\juliaup\julia-1.9.4+0.x64.w64.mingw32\bin\libjulia.dll

Cool! C:\Users\ForbearChen\AppData\Local\Programs\Julia-1.9.4\bin\libjulia.dll

[droodman]

OK. I'm not sure why it is failing to initialize Julia. But I at least want to make the simplest example I can of the problem, which I can share online. If you are willing, I'd like you to run a small program I wrote that just focuses on the issue. It is not a Stata plugin. You run it from CMD. It is called Project1.exe. It just tries to load libjulia.dll and then initialize Julia. The download link is: https://1drv.ms/u/s!Avm4GrhZKgeamf0x9_nIdfW9L3fSaA?e=YKbdCE

In general you should be careful about running programs downloaded from the Internet!

The entire code of the program:

#include <julia.h>
#include <iostream>
#include "windows.h"
int main() {
   HINSTANCE hDLL = LoadLibraryA("C:\\Users\\ForbearChen\\AppData\\Local\\Programs\\Julia - 1.9.4\\bin\\libjulia.dll");
   void (*JL_init)(void);
   JL_init = (void (*)(void))GetProcAddress(hDLL, "jl_init");
   JL_init();
    std::cout << "Didn't crash!\n";
    return 0;
}

Here is a screenshot of me running essentially the same program on my computer:

I'm interested in whether you get the "Didn't crash!" message.

[droodman]

You can see I'm discussing a similar issue with Julia experts. I hope to hear back from them soon.

[ForbearChen]

I placed this software in my desktop path, and when I ran it in the terminal, I only saw a briefly flashing window without any other prompts. Thanks for your assistance, and I hope to start using this excellent productivity tool soon.

[santiago-afonso]

FWIW I have the same issues as ForbearChen. I just ran this on Stata 17.0 MP

. net install julia, replace all from(https://raw.github.com/droodman/julia.ado/master)
checking julia consistency and verifying not already installed...

the following files will be replaced:
    c:\ado\plus\j\jl.ado
    c:\ado\plus\j\jl.plugin

installing into c:\ado\plus\...
installation complete.

. jl start
C:\Users\santi\.julia\juliaup\julia-1.9.4+0.x64.w64.mingw32\bin\libjulia.dll
start is not a valid subcommand.
r(198);

. which jl
c:\ado\plus\j\jl.ado
*! jl 0.6.0 2 December 2023
*! Copyright (C) 2023 David Roodman
*! Version history at bottom

. 

Running jl: 1+1 instantly crashes Stata.

Edit: running Windows 10 [Version 10.0.19045.3758]. Installed Julia through the Store, as directed by the ado's help.

[droodman]

Thanks @osnofas. I want to clarify. The version you downloaded from master is hacked to halt early and generate an error message, which you got.

So can you back up and tell me what happens if you do

ssc install julia, replace
ssc install reghdfejl, replace

...and then restart Stata, and try reghdfejl?

[santiago-afonso]

Thanks @osnofas. I want to clarify. The version you downloaded from master is hacked to halt early and generate an error message, which you got.

So can you back up and tell me what happens if you do

ssc install julia, replace
ssc install reghdfejl, replace

...and then restart Stata, and try reghdfejl?

I tried the two commands you instructed and then jl: 1+1, which instantly crashes Stata. The same with jl: reghdfejl.

reghdfejl returns "last estimates not found"

[droodman]

OK, I have reproduced the problem on an old laptop, and will drill into it.

@osnofas, jl: reghdfejl is not proper usage as it is a Stata command, not a Julia command. And reghdfejl by itself will only re-display previous estimation results, if any are present.

But jl: 1+1 should not cause a crash.

[droodman]

Does it work if you:

1. Install Julia from here, choosing "64-bit (installer)". During the installation process, make sure to click "Add Julia to PATH".
2. Revert to julia.ado version 0.5.8, with:

   net install julia, replace all from(https://raw.github.com/droodman/julia.ado/v0.5.8)

   Restart Stata.

[santiago-afonso]

Success. reghdfejl is blazingly fast. Congratulations!!! (I suggest replacing the install method mentioned in the help by this one.)

Is there a way to do "function masking" as in R to replace reghdfe with reghdfejl? Several other commands such as eventdd use reghdfe and could really benefit from the speedup.

(I think I'm going to edit those .ados manually anyway!)

[santiago-afonso]

(Truly thankful, I was just beggining work on a 40M rows dataset and its a PITA!)

[droodman]

Perhaps I have fixed the problem?

Can you try

net install julia, replace from(https://raw.github.com/droodman/julia.ado/v0.6.2)

and restart Stata. My hope is that this will work whether you have installed Julia from the Microsoft Store or from the direct download page.

[droodman]

@osnofas ask an ye shall receive. :-) Now you can do reghdfejl mask and reghdfejl unmask.

net install reghdfejl, replace from(https://raw.github.com/droodman/reghdfejl/v0.4.0)

[ForbearChen]

Perhaps I have fixed the problem? Can you try

net install julia, replace from(https://raw.github.com/droodman/julia.ado/v0.6.2)

I have tested this new version, and it runs smoothly on my machine. Thanks for the ongoing debugging, and once again, I appreciate your efforts. I hope it gains more users' recognition and usage!

[santiago-afonso]

Can confirm that it works. Fully uninstalled Julia and checked that no PATH environmental variables remained, rebooted, installed from MSFT Store. Now it works.

"Masking" 😂 also works. I tried eventdd, method(hdfe) and it calls Julia!!! This is great. I'd suggested talking to Sergio Correia to merge this (reghdfejl) into reghdfe as an option. Congratulations and thanks again.