]dbuild apex.dyalogbuild fails on character matrix generated from ]link.add

[bernecky]

I finally figured out how to make ]link add work: it happily wrote a character matrix to the linked folder, but the following ]dbuild failed this way:

[015] ***** ERROR ***** *** could not fix "/home/apex/apexgit/Source/Root/apexutils/PFAT.apla" The text of PFAT.apla, a 126×14 character matrix, is:

['FN'       'CT'     'ZT'     'CFUN'   'FNCL'       'LID'          'L0'    'L1'    'QCT'   'QIO'   'QPP'   'QPW'   'QRL'   'UNIF'
 'mdiam'    '?????'  '?????'  ('?'⋄)   ('?'⋄)       ('?'⋄)         ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('0'⋄)  ('0')
 'ddiam'    '?????'  '?????'  ('?'⋄)   ('?'⋄)       ('?'⋄)         ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('0'⋄)  ('0')
 'mdoll'    '?????'  '?????'  ('?'⋄)   ('?'⋄)       ('?'⋄)         ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('0'⋄)  ('
...
'cup'      '?????'  '?????'  ('.'⋄)   ('?'⋄)       ('.'⋄)         ('.'⋄)  ('.'⋄)  ('0'⋄)  ('0'⋄)  ('0'⋄)  ('0'⋄)  ('0'⋄)  ('?')
 'hydr'     '?????'  '?????'  ('.'⋄)   ('?'⋄)       ('.'⋄)         ('.'⋄)  ('.'⋄)  ('0'⋄)  ('0'⋄)  ('0'⋄)  ('0'⋄)  ('0'⋄)  ('0'⋄)]

Suggestions welcome. This version of Link is from a week or so ago, running on Ubuntu 20.10, and Dyalog APL 18.0, latest version, long ago lost in the annals of the log file.

[abrudz]

It is not clear to me how ]dbuild would end up wanting to fix a .apla file, but OK. Can you to attach the file? You should be able to drag and drop it here into a comment. Feel free to anonymise it if necessary.

Btw, you should always be able to determine your APL version with ⎕WG'APLVersion'.

[mbaas2]

This needs to be fixed in DBuild. I'll do that.

[bernecky]

Re abrudz's comment about fixing a .apla file:

I find it puzzling that Adam is puzzled about why ]dbuild should be able to act as an inverse function to ]link, in the sense that I would like to be to back up a workspace: all variables and functions in all namespaces, into a host system folder, and be able to restore the workspace from that. To me, that is a fundamental design goal of such a tool. Did I miss something there, and ]link is intended to provide only write capabilities?

Let me back up a bit here, as some background may assist in the design process. What problems am I trying to solve by using ]link, for maintenance AND distribution of the APEX compiler? [BTW, Aaron Hsu's Co-dfns compiler, does, IMO, an excellent job of packing itself so that all it needs on the APL end of affairs is a single codfns.dws workspace, placed in the MyUCMDS folder. There are no needed dinky bits to be hidden under carpets in the corner, no .xxx.rc confuseration files, etc.]

1. I want to create, edit, maintain (git), and view all aspects of APL-based code in exactly the same way that I maintain, e.g., C code. I.e., using host-based tools. I do not want to have to learn Yet Another Editor, nor do I want to know anything about, e.g., SALT. It is bad enough that I have to learn about ]dbuild files, instead of extending a Makefile.

2. I want to be able to package the compiler so that the install is as simple and trouble-free as Aaron's compiler, with no Oh You Need This Directory Full Of Other Bits required.

I was under the impression that ]link is intended to facilitate the above.

It should not, IMO, matter, whether I choose to edit a file in APL or in a host editor, such as gedit or vi. Sadly, it does. E.g., in the APEX compiler, a fair number of files (configuration, properties, code fragments in code generators) are maintained in host system text files. E.g. the Primitive Function Attribute table is maintained in an easy-to-read text file, PFAT. It looks like this, in part:

    ⍴PFAT 
126 14
      PFAT[0, 5?126;]
 FN     CT     ZT     CFUN  FNCL   LID  L0  L1  QCT  QIO  QPP  QPW  QRL  UNIF 
 dbang  BDDZC  BDDZC  .     dmisc  .    .   .   0    0    0    0    0    0    
 hydr   ?????  ?????  .     ?      .    .   .   0    0    0    0    0    0    
 mdoll  ?????  ?????  ?     ?      ?    ?   ?   ?    ?    ?    ?    0    0    
 mmpy   BIDZC  BIIZC  .     msf    ,    .   .   0    0    0    0    0    1    
 mrbr   ?????  ?????  ?     ?      ?    ?   ?   ?    ?    ?    ?    0    0    
dmpy     BIDZC  BIDZC  dmpy   dsf        1            0   .   0    0    0    0    0    1    

Yes, PFAT is really a matrix, Virginia.

Compare a simple text file to the pig-shit-ugly form of PFAT generated by ]link:

'FN'       'CT'     'ZT'     'CFUN'   'FNCL'       'LID'          'L0'    'L1'    'QCT'   'QIO'   'QPP'   'QPW'   'QRL'   'UNIF'

I find it far easier to maintain tables and code fragments, such as:

%Fragment iota x01 x bidc i NonNeg
inline int[.] $FNAMENonNeg($YTYPE y, int QUADio)
{ /* Index generator on ScalarN when N is non-negative integer */
  z = QUADio+iota(toi(y));
  return( z);
}

in my host system editor.

In the current version of APEX, these text files are host files. My goal is to move them into the APL namespaces where they are used, but still maintain their visual ease of editing as tables (albeit represented as vectors) in a host editor. Perhaps all that is needed is: )pleaseinvokemyhostED PFAT, assuming that it would preserve the ]link magic.

I want to make it as easy as possible for The Great Unwashed to be able to extend the APEX compiler as they wish, so in all of the above, when the pronoun "I" appears, please read that as "We" or "They".

BTW: ⎕wg'APLVERSION' Linux-64 18.0.39660.0 S Development

[abrudz]

I find it puzzling that Adam is puzzled about why ]dbuild should be able to act as an inverse function to ]link, in the sense that I would like to be to back up a workspace: all variables and functions in all namespaces, into a host system folder, and be able to restore the workspace from that. To me, that is a fundamental design goal of such a tool. Did I miss something there, and ]link is intended to provide only write capabilities?

I'm puzzled because Link is it's own inverse. It creates a two-way synchronisation between the workspace and the host system folder. So just as you use Link to create files and folders from workspace content, so to do you use Link to create workspace content from files and folders.

You can easily use Link to "build" a workspace from a folder of source files (without setting up a synchronisation). Simply use Import.

1. I want to create, edit, maintain (git), and view all aspects of APL-based code
   in exactly the same way that I maintain, e.g., C code. I.e., using host-based tools.
   I do not want to have to learn Yet Another Editor, nor do I want to know anything
   about, e.g., SALT. It is bad enough that I have to learn about ]dbuild files, instead
   of extending a Makefile.

Link lets you use any external editor you want; whenever a file is modified, the change is propagated into APL. You shouldn't use SALT and Link together, and as far as I can tell, you don't need anything from DBuild that Link can't do by itself.

Compare a simple text file to the pig-shit-ugly form of PFAT generated by ]link:

'FN'       'CT'     'ZT'     'CFUN'   'FNCL'       'LID'          'L0'    'L1'    'QCT'   'QIO'   'QPP'   'QPW'   'QRL'   'UNIF'

I'm not sure what is so ugly about this? How do you imagine it could be stored instead? Remember that the format has to work for all (or at least most) types of arrays, not just matrices of character vectors, so we e.g. need a way to distinguish '12' and 12.

If you prefer storing settings in a JSON, XML, or CSV file, then the corresponding system functions should make that a breeze.

[bernecky]

Sorry about the above typo in "compare a simple...". Pick later rows of the PFAT table shown earlier - the header row is degenerate.

Yes, I know that I could change things to store settings in other formats. I do not, at the moment, have either the time nor the inclination to do that, just because ]link was not working as advertised. Let me back up a bit here, and then I'll come back to to this:

In my opinion, ]link.create NS FOLDER should, BY DEFAULT, include the entire namespace tree, NS, that it is given as an argument. It should be possible to then do, e.g, tar -cvf app.tar FOLDER, copy and untar app.tar to another system, and do ]link.create NS FOLDER on that system, and have a working app on the target system. The docs, as of 2020-12-20, suggest that this may be the case, even though the SOURCE is vague on what happens when both directory or namespace are not empty. It suggests the behavior through the undefined term "authoritative"; this should be made explicit, IMO.

Which brings us back to ]dbuild not working: It should be able to operate on anything that ]link has created, I think. Why does it not do so? It defines functions and namespaces quite happily, but chokes on variables.

[bernecky]

My rationale for claiming that ]dbuild is broken for variables is that I may want to package an app that is maintained with link, but not have it include anything Linkish in the packaged copy.

[abrudz]

Since we don't have access to the raw PFAT, it is a bit hard to look at it ;-)

The reason ]dbuild doesn't understand some of the things Link creates is historical. ]DBuild predates Link. That's not an excuse for neglecting to update ]DBuild, but it does at least explain the situation.

[bernecky]

1. Don't worry about PFAT. It is simply a rank-2 character matrix.

2. Is the intent to desupport ]dbuild? If so, that's fine, but I would like to avoid having to do non-productive work more than once.

3. Unrelated: Is the ]link information maintained in the workspace? I.e., if I pick up a .dws file and move it to another system, are the ]link entries preserved? [I do not know if I want them to be preserved or not. Probably not, if I am trying to package an app for distribution.]

[bernecky]

Three problems are still unsolved as of 2020-12-27. Here is how to reproduce it, without using ]dbuild:

PFAT.TXT

1. When I added the 126-row APL character matrix PFAT this way: ]add PFAT, it generate ".apla", rather than ".aplc", so I renamed it by hand, in the hope of avoiding errors in (2).

2. )clear followed by ]link.create Source/Root creates a plethora, or in this case, 126 plethoras, of error messages:

   line(0,0) : error AC0120: assignment is not permitted outside of a :Class or :Namespace "['FN'       'CT'     'ZT'     'CFUN'   'FNCL'       'LID'          'L0'    'L1'    'QCT'   'QIO'   'QPP'   'QPW'   'QRL'   'UNIF'" 
                                                                                        ^
   line(1,1) : error AC0120: assignment is not permitted outside of a :Class or :Namespace " 'mdiam'    '?????'  '?????'  ('?'⋄)   ('?'⋄)       ('?'⋄)         ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('0'⋄)  ('0')" 
                                                                                         ^
   line(2,1) : error AC0120: assignment is not permitted outside of a :Class or :Namespace " 'ddiam'    '?????'  '?????'  ('?'⋄)   ('?'⋄)       ('?'⋄)         ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('?'⋄)  ('0'⋄)  ('0')" 
                                                                                         ^
   line(3,1) : error AC0120: assignment is not permitted outside of a :Class or :Namespace " 'mdoll'

3. The error messages do not, as far I can tell, give any information about which item (In this case, PFAT) is causing the troubles.

[abrudz]

It is not entirely clear to me what you were trying to achieve by renaming the generated file. Link relies on the file extension handle each file in the correct manner. By renaming from ".apla" to ".aplc" (the latter indicating a class), one causes Link to try ⎕FIXing the file content, which errors on every single line, them being invalid as a lines of a class definition.

[bernecky]

Ah, thanks; I was misled by this page, specifically the settings of those filename suffixes:

https://github.com/Dyalog/link/blob/master/help/Link.Create.md

Mea culpa on item 1.

It would be nice if that table had another column, giving the English name of the type, e.g, Variable, Class, etc.

I have now renamed those files from .aplc back to .apla. Errors 2 and 3 still occur.

[abrudz]

I suspect this isn't a Link problem, but rather a DBuild problem, and I understand that one of my colleagues is working on fixing it. However, the current holiday may delay a release of the fix. Meanwhile, have you tried using Link directly to reconstitute your workspace from source?

[bernecky]

Ah, I do not use ]dbuild any more, as requested. This is happening with ]link.add. UNLESS it happens that ]dbuild leaves Secret Information lying around somewhere.

The ws is built with: )clear ]link.create # Source/Root

See above from 3 hours ago, with PFAT.TXT in blue ink.

[abrudz]

I put PFAT.TXT into /tmp/linktest renamed it PFAT.apla, then:

      ]create # C:\tmp\linktest
Linked: # ←→ C:\tmp\linktest
      ⍴PFAT
126 14
      3 4↑PFAT
┌─────┬─────┬─────┬────┐
│FN   │CT   │ZT   │CFUN│
├─────┼─────┼─────┼────┤
│mdiam│?????│?????│?   │
├─────┼─────┼─────┼────┤
│ddiam│?????│?????│?   │
└─────┴─────┴─────┴────┘
      ⎕SE.Link.Version
2.1.0-beta22

[bernecky]

I am running the same version of Link as you are. However, watch this, as I think we are getting closer to the real fault:

     )clear
clear ws
      PFAT←⎕fread 1 1 
      ≡PFAT
2
      ]link.create # /tmp/crud 
Link Warning: ⎕SE.Link.U.UCMD_Run: .NET or .NetCore not available - Link cannot watch directory
Linked: # → /tmp/crud
      ]link.add PFAT
Added: #.PFAT
      PFAT≡ ⎕fread 1 1
1
      )clear
clear ws
     ⎕sh'ls -l  /tmp/crud'
total 20
-rw-rw-r-- 1 apex apex 18555 Dec 27 15:09 PFAT.apla
      ]link.create # /tmp/crud 
Link Warning: ⎕SE.Link.U.UCMD_Run: .NET or .NetCore not available - Link cannot watch directory
Linked: # → /tmp/crud
      PFAT≡ ⎕fread 1 1
0
      ≡⎕fread 1 1
2

      ≡PFAT
¯2  

    ⎕size 'PFAT'
83800
      xx←⎕fread 1 1
      ⎕size 'xx'
84792

Note that the PFAT from ]link is NOT the same as the one we started with. The ]link PFAT is also a bit tinier than the one we started with. [Type squeezing, maybe?] The depth of the new array is negative, indicating that it is not of uniform depth, whereas the original (⎕fread 1 1) is of uniform depth.

[abrudz]

Would it be possible for you to attach that component file for experimentation?

[bernecky]

Sure!

crud.dcf.txt

I had to append ".txt" to the filename to make github happy.

[nicolas-dyalog]

Not a link issue - is a DBuildTest issue handled by Michael as Dyalog/DBuildTest#5