Folding feature request

[jeff-bowles]

Requesting the ability to flag certain record types as "fold begin" and "fold end" points

Most of my data files are tens of thousands of records long, consisting of contiguous and logically-related groups of records I call "record sets". It would be very helpful if certain record types could be designated as "fold begin" points, while others could be "fold end" points.

Why the standard "user defined language" feature of N++ doesn't suit my need I have tried using the standard "user defined language" feature of N++ to designate a pattern that identifies "fold begin", but I've not found way, with the type of data files I use, to reliably indicate a "fold end". Mainly, this is because the logical groups of records do not always end with the same record type.

Proposed solution For this reason, it would be a wonderful feature if FWDataViz could allow the "fold begin" point to also be the "fold end" for the prior logical group.

A little background info I posted a question in the N++ forum, but got a disappointing answer that the UDL feature has not been updated in a long time and my request would be behind many others in priority.

Examples and (I hope) clarifications

Here are some examples of typical "record sets" using common metasyntactic symbols.

Example 1 - a logical record set

foo record
bar record
baz record
plugh record
qux record

Example 2 - a logical record set

foo record
bar record
baz record
plugh record
qux record
fred record
waldo record

Example 3 - logical record set

foo record
bar record
baz record
plugh record
fred record

In all examples, "foo" is the first record of my record set, so I would like to be able to check a box on the FWDataViz metadata editor indicating "this is a fold-opening record type".

For a "waldo" record (Example 2), I could check a different box indicating "this is a fold-closing record type". In my world, "waldo" always marks the end of a logical record set, if present.

Conversely, I would not declare "qux" or "fred" as fold-closing types, because they are not always present, nor are they always the final record type in a set.

The real magic would be the additional ability to designate "foo" records as being both a fold-close (ending prior record set) and a fold-open point (beginning next record set). Maybe we call this a "open implies close" feature. I don't know if special handling would be needed at BOF and EOF, but I would not care so much if the top and bottom record set in a file were broken / did not fold correctly, as long as the thousands in between worked.

Sorry for the long-winded explanation. If there is any interest in adding this feature, I'd be happy to provide more examples or clarification of the problem / proposed solution.

Thanks for an already-awesome plugin!

Jeff

[shriprem]

Hi Jeff, it's nice to be getting good enhancement ideas from you again!

I totally agree that Foldable Record Blocks will be an excellent feature to have in a data-viewer plugin such as FWDataViz. But there are some technical issues which will need to be resolved to implement this feature.

Firstly, you may be aware that FWDataViz visualizes only those lines that are visible on the screen at a time. Due to this approach, the plugin does not suffer lags even with large files.

However, the Foldable Record Blocks feature could require a full scan of an entire file to identify the start and end points of all the record blocks in the file. My initial assessment is that this is doable since the currently available Data Extraction feature in the plugin is doing a full file scan as well.

Then there are some Scintilla & Notepad++ API issues that I need to resolve to implement folding without being a lexer since FWDataViz is not a lexer. Then, finally, I need to provide interface controls within the plugin to define and optionally activate the record folding structure. All this will take quite some time to implement. I will keep you posted.

[jeff-bowles]

Thank you! All my enhancement ideas come from tedium doing my job. I'm happy to keep them coming for you!

[shriprem]

@jeff-bowles wrote:

Hmm - maybe a good clue for you. The "toggle line" button in the FWDataViz panel will toggle the folding on a node without making all the fold points disappear. It will toggle back (unfold) correctly as well.

Regarding the lack of colorization upon folding via NPP menu, I have documented this as a known issue. See the first item below. Source: VersionHistory.md

Known Issues

1. If you use Notepad++ menu options to fold levels, the text in the view page may end up partially colorized. However, if you move the caret (i.e., keyboard cursor) to a different position, the page will be fully colorized again. You can avoid this by using the Fold and Unfold buttons on the FWDataViz side panel.
2. If you edit the data file, you may also need to click on the Apply button on the FWDataViz side panel to rescan the file and rebuild the new folding structure. No workaround for this, since any file edits may require a full file scan which will not be optimal if done automatically for each bit of file edit.

[jeff-bowles]

My issue was slightly different. Although the colorization behaves as you described in "known issues", I was discussing folding only. Perhaps they are the same underlying issue, because the side panel buttons do work correctly.

[shriprem]

@jeff-bowles wrote:

Now I see that MultiLineGSN.txt is not defined for folding. Please disregard.

I tested with IPAC_ANSI.txt and got the same behavior as icd10cm_order_2020.txt; that is where the fold points disappear upon clicking any one of them. I also observed that switching to a different document (via the NPP tabs) and back will restore the appearance of the fold points, as will File --> "reload from disk".

I suspect that the match-up of File Types is failing between the default_Visualizer.ini and default_FoldStructs.ini files.

If not yet modified from the file versions I provided you in the zip file, the default_FoldStructs.ini will have these lines:

...
FileType=FT008_ICD-10_BILLABLE-FLAGGED_ORDER_CODES
FileLabel=☣ ICD-10 Billable-Flagged Order Codes
...
FileType=FT010_TREASURY_IPAC_FILE
FileLabel=Ŧ Treasury IPAC File
...

The default_Visualizer.ini should have sections with exact matches to the FileType values in the default_FoldStructs.ini file. Like this (in the default_Visualizer.ini file):

...
[FT008_ICD-10_BILLABLE-FLAGGED_ORDER_CODES]
FileLabel=☣ ICD-10 Billable-Flagged Order Codes
...
[FT010_TREASURY_IPAC_FILE]
FileLabel=Ŧ Treasury IPAC File
...

When I say exact match, the numeric sequence that follows the FT prefix should match across both files. For the Billable-Flagged Order Codes, it is FT010 in the listing above. For the Treasury IPAC file, it is FT012. Please check if the two values match between the two files on your system. If they don't, edit the default_FoldStructs.ini file values to match the default_Visualizer.ini values. Not the other way around. (The FileLabel value is just there for info. A match on that is not needed between the two files.)

Another point to note:

If you are loading the sample files into NPP via the plugin's Sample Files side-panel button or plugin menu option, then the plugin will use the default_Visualizer.ini and default_FoldStructs.ini files located in the <Notepad++_executable_folder>/plugins/FWDataViz/ folder.

However, if you are loading the sample files into NPP by other means, then the plugin will rely on the Visualizer.ini and FoldStructs.ini files located in the %APPDATA%/Notepad++/plugins/Config/FWDataViz/ folder for non-portable installs of NPP and <Notepad++_executable_folder>/plugins/Config/FWDataViz/ folder for portable NPP installs. Then the FileType value matching instructions I have provided above should be applied to these two latter files instead.

As I indicated in my usage notes earlier, the folding involved in the Ouroboros.txt file is more complex than these two sample files. So, there is no reason the folding feature would fail on these two instead of Ouroboros.txt. Appreciate your time.

[shriprem]

@jeff-bowles , after you return from travel, please provide info on the Windows OS version, and the Debug Info from your Notepad++. Please copy and paste the info shown in the Debug Info dialog when you click the ? menu and then Debug Info....

Meanwhile, I will test this plugin on Windows 10 and Windows 7 PCs. I am currently using Windows 11. The plugin's fold feature is working fine for me with Notepad++ 8.4.1, NPP 8.4.2, and the upcoming NPP 8.4.3 pre-release versions.

[jeff-bowles]

Here are my "debug info" details. Windows information is included. Please let me know if you need something additional.

• Notepad++ v8.4.2 (64-bit)
• Build time : May 29 2022 - 16:47:30
• Path : C:\Program Files\Notepad++\notepad++.exe
• Command Line : $COMMAND_LINE_PLACEHOLDER$
• Admin mode : OFF
• Local Conf mode : OFF
• Cloud Config : OFF
• OS Name : Windows 10 Enterprise LTSC 2019 (64-bit)
• OS Version : 1809
• OS Build : 17763.2928
• Current ANSI codepage : 1252
• Plugins:
  • ComparePlugin (2.0.2)
  • FWDataViz (2.6)
  • HexEditor (0.9.12)
  • mimeTools (2.8)
  • NppConverter (4.4)
  • NppExec (0.8)
  • NppExport (0.4)
  • NppScripts (2)
  • PreviewHTML (1.3.2)

[shriprem]

I have tested the plugin on Windows 7, Windows 10 and Windows 11 PCs; with 32-bit and 64-bit versions of NPP; in full-install and portable modes. With all these combinations, I used the same set of plugins you had provided in your NPP Debug Info. I am still unable to find any issues with the folding feature of the plugin.

That said, I carefully re-read your issue descriptions and may have a resolution to your issue. Firstly, here is my analysis.

Notepad++ typically uses three margins: one to display line numbers, second to display bookmarks & hidden line indicators, and the third to display folding indicators.

When you click on one of the "fold points", something is causing this (third) folding margin to be hidden; and the fold margin is getting restored when you either switch document tabs or reload the file. This may be happening either due to the settings deep inside one of the other plugins or due to some confluence of factors on your system interacting with the FWDataViz plugin.

As you already know, FWDataViz plugin is technically not a lexer. Maybe one of the other plugins is sensing that the fixed-width document with the folding points is not a lexed document, and so is preemptively hiding the fold margin as soon as you click on any folding point in that document. If this is the cause, we can easily isolate that plugin. But before we need to go about isolating that plugin, you should test again with the new DLLs I have attached here: FWDataViz_x64_2.6.0.0-July5.zip FWDataViz_x86_2.6.0.0-July5.zip

Only the FWDataViz.dll files have changed in the zip files as compared to the version I had provided earlier in the PR comments. Close NPP before you copy this file. After you copy the new 64-bit DLL file into your NPP plugin folder, the file contents of that folder should match this: [_Ignore the NPP_Plugin_Darkmode.dll file. It is not used anymore. It may or may not be present in your installation. If you do have it, it can be safely deleted._]

After you launch NPP, confirm that you indeed have the new plugin version by clicking the menu option for Plugins » Fixed-width Data Visualizer » About. You should see:

This version fixes couple of issues arising from the fold margin preference settings in NPP.

1. This build fixes the incorrect indicator issue if 'None' was selected for the Fold Margin Style.
2. It also fixes the issue arising when the bookmarks margin is hidden.

When you test, please click on the Sample Files button or menu option to open the sample files marked as 'Foldable'.

Note: If you try opening the icd10cm_order_2021.txt file directly, it will be auto-detected as the non-foldable version of the same file under Sample Single-Record-Type Files.

Also, please confirm if the sequence numbers are being displayed as per your other enhancement request in the Jump To Field dropdown list. Thank you!

[jeff-bowles]

Shridhar,

First some good news - field sequence numbers are displayed in the Jump to Field dropdown list!

Now the rest. Mostly, the "bugs" I've reported previously are still present in the July 5 version of the DLL. It occurred to me that perhaps my expectations are not correct about how the plugin should work.

My settings First, I'll say that I matched my Margin/Folder/Edge settings to your screen shots. I had previously been using the "box tree" folder margin style, and had both line numbers and bookmarks margins enabled, although my line numbers had been set to "constant width". Also, my "vertical edge settings" had been a blank box, but now I match your settings of "120 150 200".

Mutual understanding Next, I think I should clarify my terminology on what I mean by the folding symbols "disappearing". At no point does the allocated space for the +/- symbols go to zero width; instead, I mean that the space becomes devoid of visible characters.

Also, I'm aware of the difference in expected behavior between loading a sample file via the FWDataViz samples menu and simply loading a file from the NPP file menu. In all cases when I'm discussing sample file behavior, be assured I loaded the file through the plugin menu.

Reality Check I should say that I know what a lexer is, and that FWDataViz is not one. Perhaps this feature request is not feasible under that constraint. My programming language of choice is Delphi (Pascal) and I do some work in c#. However, I'm a complete novice in how plugins interact with NPP / scintilla, so perhaps my concepts of how something should work is incompatible with the architecture.

Testing aids For beta testing, it would be good if the FWDataViz panel had a group box indicating the currently loaded ini file paths so we can be sure that FWDataViz "sees" the files we intended. We could use this information to know whether a file was loaded via the samples menu, or where exactly NPP believes its config files are located.

With the understanding that this might be too slow for "production", another useful testing aid would be a way of forcing the plugin to pre-scan the entire document file and place whatever markers are necessary for folding somewhat permanently. In other words, allow for the elimination of the need for the "apply" button until beta testing is done.

Testing results

Here are some of my testing observations. Some of these may be "as designed" or limitations of the architecture.

1) The display of folding points is not stable, even without clicking the folding node indicators or clicking the fold buttons. Simply scrolling to the bottom of a ICD10 file, for example, and back to the top can result in previously displayed fold points being blank. I can bring them back with the "apply" button. I'm not sure if this is because of the strategy of only visualizing the small buffer in the current viewport, or if it is an actual bug.

2) I cannot seem to get FWDataViz to "see" my INI files to define folding for my own file types. The "testing aid" above might help there. However, I think making the plugin work 100% for the sample files should be a priority over getting my own custom files to work.

I'm happy to continue interacting with you and testing as long as you want.

Jeff

[shriprem]

Jeff, I always test my code and share it with others only when I am fairly confident that it will work as advertised. I do not wish to waste everyone's time with poorly written, unstable code. I am unable to replicate or mentally visualize the issues you have been describing. Responding with long messages on GitHub takes several hours of my time. I think it would be more efficient if I can establish a more direct (and real-time) channel of communication.

I only had to enter your name in my LinkedIn page and your profile picture popped up as the very first search result. (I know it is you since the profile picture matched the one with your posting at the NPP bulletin board.) Please accept my connect request at LinkedIn. Thank you.

Meanwhile, I will try providing a point-by-point response to your recent feedback in the post to follow a bit later.

[jeff-bowles]

By the way - I'm in "UTC - 4" time zone,

[shriprem]

Jeff, Today, we have both confirmed that the folding features are working well in the portable install of NPP but not in the default full-install version on your PC.

So, I would like a hands-on look at the files in this latter installation. Could you please zip the C:\Program Files\Notepad++\ folder tree in your PC and provide it to me? You can upload it here or email it to me. Thank you!

By default, there shouldn't be any private/sensitive files under the C:\Program Files\Notepad++\ folder. But please double-check for that if you are uploading the zip file here.

[shriprem]

The NPP zip file may not be needed. Instead, the issue may lie with the settings in the AppData\Roaming\Notepad++ folder. So, try this:

1. Close NPP.
2. In the File Explorer address bar, type %APPDATA% and hit ENTER. When you do this, you will navigate to the Appdata\Roaming\ folder for your user profile in Windows. Something like: C:\Users\jeff-bowles\AppData\Roaming\.
3. In this AppData\Roaming\ folder, rename the Notepad++ subfolder to something different, for example: _bkup_Notepad++.
4. Get the FWDataViz.dll from the latest commit. Page link. This DLL has changes from these following commits since the last version of DLL you got 2 days ago :
   • Added wait cursor for long operations
   • Optimization of Fold-related file events.
   • Debug Info button on the panel for displaying list of active INI files
   • Specified Calltip Font Name, Weight, Fore & Back colors.
   • Removing BOM from INI files if encoded as UTF8-BOM
5. Relaunch NPP. (At this point, a new version of the AppData\Roaming\Notepad++\ folder will get created automatically.)
6. Try the folding feature with the sample files of FWDataViz. If it works fine:
   • The issue is with one of the files in the AppData\Roaming_bkup_Notepad++\ folder you backed up in step 3.
   • Please zip the AppData\Roaming_bkup_Notepad++\ folder and email it to me.

Thanks!