memory and array

[JoseQuintas]

1 million elements, 100MB each one. 100TB String, NO ERROR Seems that I can expand array size with no limit. harbour 3.2 32 bits, on windows 10 shows 100MB of use. using a single string var, limit exists.

#include "inkey.ch"
#include "hbgtinfo.ch"

REQUEST HB_CODEPAGE_PTISO

#define THISTEST 0

PROCEDURE Main

   LOCAL nKey := 0, x, t := 0, i

   SET EVENTMASK TO INKEY_ALL - INKEY_MOVE
   Set( _SET_CODEPAGE, "PTISO" )

   SetMode(36,132)
   SetColor( "W/B" )
   CLS
   ? Memory(THISTEST)
   x := Array(1000000)
   AFill( x, Space(100000000) )
   ? Memory(THISTEST)
   FOR EACH i IN x
      t += Len( i )
   NEXT
   ? Transform( t, "999,999,999,999,999" )
   hb_MemoWrit( "um.x",x[1] )
   ? Memory(THISTEST)

   DO WHILE nKey != K_ESC
      nKey := Inkey(0)
   ENDDO

   RETURN

[snaiperis]

A single call to SPACE() performed. Same SPACE(100MB) result string filled the whole array.

[JoseQuintas]

And how explain about to use 1TB space, if not available memory?

[snaiperis]

You text was "windows 10 shows 100MB of use", not 1TB. I hope 100MB is available on your Win10 machine.

[JoseQuintas]

Please calculate string size use. 1 million X 100MB = 100 TB, this is total string size 100 TB How Harbour manage this ? How 100 TB uses only 200MB memory? How 32 bits manage 100 TB ? How available memory is allways the same, no matter about array size ? It is ok ? or it is a dangerous bug? If this is true, than I can use a CD of 600MB, save memory content, and I will have 300 TB, 500 times real size of CD. Do you understand my question about memory and string size? A UNIQUE ELEMENT OF ARRAY HAVE 100MB, there are 1 million elements of 100MB, 1 million of strings of 100MB previous information about 1TB was wrong, because I change sample source code and do not update this number.

[JoseQuintas]

20/09/2021  11:50       193.893.864 OpenOffice41.exe

#define THISTEST 0
? Memory(THISTEST)
   x := Array(10000000)
   AFill( x, MemoRead( "openoffice41.exe" ) )
   FOR EACH i IN x
      t += Len( i )
   NEXT
   ? Transform( t, "999,999,999,999,999,999" )
   ? Memory(THISTEST)

  2097151                
 1,938,938,640,000,000   
  2097151                

[snaiperis]

Please calculate string size use. 1 million X 100MB = 100 TB, this is total string size 100 TB

Wrong calculation, should be: 1 x 100MB = 100MB, because single SPACE() call performed.

How Harbour manage this ? How 100 TB uses only 200MB memory? How 32 bits manage 100 TB ?

Does not have sense for 100MB.

How available memory is allways the same, no matter about array size ?

Yes, if it is the single string.

It is ok ? or it is a dangerous bug?

Yes. No.

If this is true, than I can use a CD of 600MB, save memory content, and I will have 300 TB, 500 times real size of CD.

You never had 100TB, so, you'll never have 300TB.

Do you understand my question about memory and string size?

Yes.

A UNIQUE ELEMENT OF ARRAY HAVE 100MB, there are 1 million elements of 100MB, 1 million of strings of 100MB

There are no 1 million elements (strings). There is a single one string.

For better understanding TEST1. How many elements has array a? a := ARRAY(1000000) AFill(a, {"A"}) ? a[1, 1] ? a[2, 1] a[1, 1] := "B" ? a[1, 1] ? a[2, 1] // Why "second" element is "B" if I changed the "first" element?

For better understanding TEST2. How many allocation were made? 1? 1000000? a := ARRAY(1000000) AFill(a, MySpace(100000000))

FUNC MySpace(n) ? "Allocate", n RETURN Space(n)

[JoseQuintas]

And that other test with open office ? Fill 10 million elements with near 200mb. no error. 10 million copies of 200MB is normal? near 2.000 TB of string and no error. Check the sum of the len of each element after fill array.

[snaiperis]

I did my best. Sorry. Ask for explanation on user forum.

[Bacco]

For anyone interested, I provided some explanation here:

https://groups.google.com/g/harbour-devel/c/49XhWTupyFM/m/05E7HfvAAQAJ

TL/DR:

Harbour has a kind of Copy-on-write mechanism, so when you create a string and do an AFill or anything that produces the exact same string (a simple cTwo :=cOne is enough), both will reuse same memory data until you change anything. So in the example above, theres only one copy of the file in memory, and 1000000 strings that use the SAME memory data, so there's no overflow.

Important: don't confuse it as a variable "by reference". When you have variables by reference, you change cTwo, the change will reflect in cOne. In this case both are independent strings. If you change cOne (or cTwo), internally Harbour will allocate space for both strings and change only one.

Please, read more detailed explanation (with a .prg code showing more details) in the link provided above.

It's not an issue, it's just a normal feature.