[Feature]: Compatibility with SolarSoft IDL

[AvijeetPrasad]

Description

In Solar physics, we usually run IDL under the SolarSoft environment, which automatically downloads and sets the paths for various instruments specific to the field. The steps for SolarSoft IDL installations are given here: https://www.lmsal.com/solarsoft/.

Typically, this means that we start IDL with the following csh environment variables, which are added when we source the ssw setup script:

#!/bin/csh
setenv SSW /usr/local/opt/solarsoft
setenv SSW_INSTR "gen iris ontology sdo hinode sot hmi aia hessi"
setenv ssw_email_address 'myemail@gmail.com'
setenv IDL_DIR /usr/local/harris/idl89
setenv IDL_DLM_PATH "<IDL_DEFAULT>:+/usr/local/IDL"
setenv PATH $PATH\:/usr/local/codes/
setenv IDL_PATH "<IDL_DEFAULT>:+/usr/local/codes/idl"
setenv IDL_STARTUP /usr/local/scripts/idl_startup.pro
source $SSW/gen/setup/setup.ssw

Note that the environmental variables are defined in csh and not bash. Usually, we do this by incorporating the above line in our cshrc file, and then start idl from that terminal session. I wonder, if similar functionality can be incorporated under the VS Code IDL extension.

Why it Matters

This would be very useful for all solar physicists, who extensively depend on the SolarSoft IDL packages for working with data from different instruments.

Suggested Behavior

One way to do this could be by defining all the environamental variables in a separate file which can be accessed by the extension. Also an option to source additional scripts would be needed to setup the ssw internal paths.

Alternate Behavior

No response

[znorman-harris]

Does this mean you have tried to start IDL within vscode and it doesn't have all the right variables to run your code?

If you launch vscode from a terminal window, it should inherit the environment varibles, which also get passed to IDL. The only one that we do tweak is the DLM path when not on Windows.

[AvijeetPrasad]

I can add all the environment variables to the extension preferences. The issue is that I need to run the line source $SSW/gen/setup/setup.ssw, which then gives the output Type <sswidl> to start SSW IDL. And only when I type sswidl, IDL starts with the ssw packages loaded. I don't know how to reproduce these steps within the extension environment.

[znorman-harris]

So, just to make sure I understand your use case correctly, you start "sswidl" instead of "idl"?

[AvijeetPrasad]

Yes. Typing “sswidl” starts IDL with all the solarsoft paths and packages loaded. See the following link for more details: https://www.lmsal.com/solarsoft/ssw_setup.html

[znorman-harris]

Ok, thanks! Do you know what the command actually does? I was chatting with the IDL team and we were wondering if you can use environment variables instead of a different command to start IDL.

The biggest challenge is that we currently have a cross-platform, and tested, way to start IDL. We don't think that it should be exposed through VSCode to change that at this point in time.

If you're the creator of this lib, maybe we can help get another way to run everything in VSCode using IDL's standard extension/customization points.

[AvijeetPrasad]

I can check this tomorrow and let you know what the ssw setup file is doing. I am not the creator of this package. It’s maintained by NASA, LMSAL etc, I think, as it supports libraries for their instruments.

One possibility would be that the notebook can allow running shell commands through magic methods (similar to what we do in Python Jupyter notebooks). That way, one can run these setup steps in the notebook before launch IDL through a shell command manually, the way we do in terminal?

[znorman-harris]

That would be great! If you also have a link to the source code, we can take a look and see what it is doing.

If it's a shell script, hopefully it's just the environment that's getting set up and we can figure something out for the extension.

[AvijeetPrasad]

To my understanding, I see that the e ssw setup file is setting up a large number of paths. Is there a way I can share these files with you? GitHub is not allowing me to add these files here in the comments.

[znorman-harris]

If the source code isn't already on GitHub, then you can zip them up and send me an email at zachary.norman@nv5.com and I can take a look.

Thanks for your help with this!

[AvijeetPrasad]

I have shared a folder ssw.zip through Dropbox to your email address. The file to look would be ssw/gen/setup/setup.ssw. In line 284, you will see that sswidl is aliased to ssw/gen/setup/ssw_idl. And going in to that file shows all the paths and variables that the code is trying to setup.

If necessary, I would suggest that after the download, you can try to run the setup as outlined here: https://www.lmsal.com/solarsoft/ssw_setup.html

Pleae let me know if you need any more information.

[znorman-harris]

Taking a look at the ssw_idl script, if you started VSCode (using the code command) instead of IDL on line 316:

It looks like that script really doesn't do much besides find IDL and pick the right startup command for IDL. If you started VSCode instead of IDL, you should get an instance of VSCode will the same environment and paths setup. Then you can let our extension manage finding IDL, starting it, and pass in the environment to the IDL process so startup files and anything else come through as you might expect.

Maybe an alternate solution here is to create another shell script called ssw_code, or something similar, so that it simplifies the logic of the shell script to not care about IDL and just starts VSCode for you while preserving the existing user experience.

[AvijeetPrasad]

Thanks. This seems to be working! It is now recognising solarsoft routines :)

I am still having some issues when trying to run on of the ssw plotting routines. See the error below:

savfile = 'AR11166_09_03_2011_h2236_hmi.sharp_cea_720s_bnfff_hmi.sav'
restore,savfile
help, bz
BZ              FLOAT     = Array[733, 382]
plot_image, bz
% Not a legal system variable: !ASPECT.
% Execution halted at: PLOT_IMAGE        244
 /astro/local/opt/solarsoft/gen/idl/image/plot_image.pro

Typically this command will open a new x-window when the plot. Similar to the tvscl routine. Do you have any suggestions on how to fix this?

[ct6502]

Hmm, that error about "!ASPECT" seems strange. I don't see that in any of our IDL codebase, so I think that must be a custom system variable that is created by one of the solarsoft routines. Perhaps a file isn't getting restored, or perhaps some "setup" script isn't getting run that should be run. If you search across the solarsoft code maybe you could locate where !aspect is being defined (search for "defsysv"). Hopefully you can then figure out why that code isn't being run. Please let us know what happens. Thanks!

[AvijeetPrasad]

Hi. Thanks a lot. Yes, these system variables are defined in the file ssw/gen/idl/system/add_psys.pro, which looks like the following:

pro add_psys

defsysv,'!image',exists=defined
if not defined then imagelib
defsysv,'!aspect',exists=defined
if not defined then devicelib
defsysv,'!debug',exists=defined
if not defined then uitdblib

return

end

Based on the suggestion here: https://github.com/interactive-data-language/vscode-idl/issues/26#issuecomment-2070378983 I replaced

# call IDL with any arguments
onintr -                                # allow ctrl c in idl
 $IDL_DIR/bin/idl $command              # no alias (see comment) slf 30-Jul

with

# call IDL with any arguments
onintr -                                # allow ctrl c in idl
# $IDL_DIR/bin/idl $command             # no alias (see comment) slf 30-Jul
 code .

I wonder, if this is the correct way to do this. I can see that the ssw routines are now added to the IDL PATH, but maybe some routines which were being compiled previously are not being done here. Do you have any suggestions on how I can check that.

[ct6502]

Yes, I think you're right about a missing command. That $command probably contains the name of a startup routine that runs a bunch of stuff, including that "add_psys". You might be able to just define an IDL startup file that would run automatically. To do that, either look at the $command variable and see what it contains, or see if you already have an IDL_STARTUP environment variable. If you do, just add the "add_psys" call to the end of it. If you don't have an IDL_STARTUP environment var, then create a brand new file, with just a single line in it: add_psys Save the file, then define the IDL_STARTUP variable to point to that file's location.

Note that "add_psys" might be just one of multiple startup files that ssw needs to run. In that case, you'll have to find the top-level IDL procedure that runs them all, and put that procedure name in your startup file instead. I hope this all makes sense. And thanks for helping to experiment with all of this! If we get this working for you, then it will help all of the other solarsoft users who want to do the same thing.

[AvijeetPrasad]

@ct6502 Thanks for your reply. I added imagelib and devicelib routines to my IDL_STARTUP file, which should add the system variables like !ASPECT and !IMAGE, but the issue persists. I am not sure what else to try. In case you would like to test it locally, I can share the ssw library folder with you. Please let me know.

[ct6502]

Interesting. Maybe put a command in your startup file, like "PLOT,[0,1]". That way you can confirm that the correct startup file is being processed. You could also try putting a "PRINT, !ASPECT" at the end of that startup file, again to confirm that IDL knows about the variable at startup time. If it does know about the !aspect, then I'm really not sure what is going on. If IDL does not know about !aspect, then you will need to figure out why those other startup routines either are not being run, or why they aren't doing the right thing.

[AvijeetPrasad]

I tried adding print statements to the startup file, and that works as expected when I run IDL separately. I also see that it shows that imagelib and devicelib were compiled. But somehow, these are not inherited in the notebook session.

[ct6502]

You're right - I just tried it in notebooks and it is not processing the startup file. @znorman-harris it looks like the IDL startup file isn't being processed with Notebooks. Is this expected behavior, and can we fix it? :-)

[AvijeetPrasad]

Thanks for the confirmation! Also, is it possible to compile routines within the notebook? That way I can try to manually compile these routine as a temporary workaround.

[znorman-harris]

Let me try to reproduce this. As long as you have the IDL_STARTUP environment variable set/defined before VSCode starts, you should be good. I'll make sure that's the case.

Thanks for the confirmation! Also, is it possible to compile routines within the notebook? That way I can try to manually compile these routine as a temporary workaround.

And yeah, you can just call the routines as procedures or functions and IDL should automatically find, compile, and run them as long as they are on your path.

[znorman-harris]

At least on Windows this is working as expected. I double checked the logic for starting and we don't check for or manipulate IDL_STARTUP in any way.

Here I have a startup file on the left that defines the variable foo and, when I execute help, foo from a notebook it shows the right value:

Also in the screenshot, I have the environment printed out in a terminal window below. Maybe check from a terminal in VSCode that IDL_STARTUP is defined as you would expect it to be? I'm not sure if this gets weird because there might be some shell startup scripts.

I guess the best way to check this would be running this from a notebook:

print, getenv('IDL_STARTUP')

[ct6502]

Ah ha! In my case, I had both IDL and ENVI installed. So the IDL notebook was starting up the IDL within the ENVI directory, but I had specified a startup file for my "normal" IDL (within the IDL directory). I went into the extension settings (click on the IDL icon on the left side of VS Code, then click "Open Extension Settings". Then scroll down until you see the setting for IDL Directory, and confirm that it is pointing to the IDL that you expect it to point to.

If you change that setting, be sure to restart VS Code so it will pick up the new IDL. You can also double check this within the Notebook by typing: print, !dir

[znorman-harris]

If you change that setting, be sure to restart VS Code so it will pick up the new IDL.

You'll also need to make sure that you manually stop and restart IDL for notebooks if you manually change the environment within the extension settings. You can do this with the buttons in the top-right of the notebook next to where it says "IDL"

[AvijeetPrasad]

Thank you for your replies. I was able to make it work by adding IDL_STARTUP variable in the extension settings. Earlier I had only defined it in my shell but not in extension settings. So, now it is running the devicelib and imagelib routines on startup and the plot_image ssw routine seems to run without any issues :)

On a separate note, is it possible to define a procedure in a notebook cell, run it and then execute it later (similar to python functions in jupyter notebooks)? For me its not working this way. I guess, running a cell is not the same as compiling it?

[znorman-harris]

You can define procedures and functions within notebook cells and use them anywhere else.

I would recommend checking out the example IDL notebook which covers the basics of cell execution, including how to create programs. You can find that here in the extension:

[AvijeetPrasad]

Thank you very much! I am able to compile and run the procedures now :)

One last query! The notebook seems to hang when a line requires user input. For example:

; Define B as a string before reading:
B = ''
; Read input from the terminal:
read, B, prompt = 'Enter Name: '

I guess it is expecting an input through the terminal or some pop-up, but I don't seem to have this option to input. Can you please have a look?

[znorman-harris]

At least for notebooks, I doubt that we will ever add support for the read procedure since we don't have a way to provide native input through VSCode or the notebook interface.

When running IDL without notebooks, we use the debug console so there is a dedicated place for users to specify input. However, read is not supported there right now because we have no way to tell when IDL needs input from read.

For the debug console, we are tracking it here.

If you are wanting to add interactivity to your notebooks, then I would maybe set aside a cell at the top for any variables that a user needs to set/update in order for the rest of the cell to work. This follows how I use them and how I make most of my automated IDL scripts as well.

[AvijeetPrasad]

Thank you for the suggestion. I wonder, can't we provide input in a way similar to how we do it for Python Jupyter notebooks in VS code? In that case, a pop-up will appear in the command bar, and we can enter the text.

In any case, feel free to close this issue, as the original ssw issue seems to be solved for the moment.

[znorman-harris]

Since I think your main issue is resolved, I'm going to close this. Feel free to use discussions or issues in the future if you have something else you want to chat about!

[steinhh]

Hi @AvijeetPrasad, Stein (also from ITA, room 214 ;-) here. We've had the same issue, our solution was to create a tiny [t]csh script, basically two lines of "sswidl", then point the extension to that script for startup. Seems to work well.