How to save NdArray object to local disk?

[mullerhai]

Hi: when I generate NdArray obj from spark dataframe ,I want to save the NdArray obj for next model training，but I don't know how to save it.

[mullerhai]

java.io.NotSerializableException: org.tensorflow.ndarray.impl.dense.DoubleDenseNdArray
  at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
  at java.base/java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
  at java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
  at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
  at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
  at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)

[karllessard]

Unfortunately there is no endpoint out-of-the-box for serializing/deserializing an NdArray. It would be a great addition though. You probably want to save the array type, shape and data instead of an object?

Can you show me how you initialize your NdArray? That might give me some cue on how we can tackle this.

[mullerhai]

Unfortunately there is no endpoint out-of-the-box for serializing/deserializing an NdArray. It would be a great addition though. You probably want to save the array type, shape and data instead of an object?

Can you show me how you initialize your NdArray? That might give me some cue on how we can tackle this.

thanks, let me see. Now we have three ways to generate NdArray

1. from java& scala normal array

val testMatrix1 = StdArrays.ndCopyOf(Array[Array[Int]](Array(1, 2, 3, 4, 45), Array(2, 4, 6, 8, 10), Array(3, 6, 9, 12, 15), Array(4, 8, 12, 16, 20)))
    val testMatrix2 = StdArrays.ndCopyOf(Array[Array[Int]](Array(1), Array(0), Array(1), Array(1)))

2. from data disk file like csv or zip file or inputStream , nio Databuffers

   
   @throws[IOException]
   private def readArchive(archiveName: String) = {
   //    val dataset =classOf[MnistDataset].getClassLoader.getResourceAsStream(archiveName) //NullPointerException
   val dataset = new java.io.FileInputStream(archiveName)
   
   val gzipInputStream = new GZIPInputStream(dataset)
   val archiveStream = new DataInputStream(gzipInputStream) //new GZIPInputStream(new java.io.FileInputStream("src/main/resources/"+archiveName))
   //      )
   archiveStream.readShort // first two bytes are always 0
   
   val magic = archiveStream.readByte
   if (magic != TYPE_UBYTE) throw new IllegalArgumentException("\"" + archiveName + "\" is not a valid archive")
   val numDims = archiveStream.readByte
   val dimSizes = new Array[Long](numDims)
   var size = 1 // for simplicity, we assume that total size does not exceeds Integer.MAX_VALUE
   for (i <- 0 until dimSizes.length) {
     dimSizes(i) = archiveStream.readInt
     size = size * dimSizes(i).toInt
   }
   println(s"size  ${size}")
   val bytes = new Array[Byte](size)
   archiveStream.readFully(bytes)
   NdArrays.wrap(Shape.of(dimSizes: _*), DataBuffers.of(bytes, true, false))
   }

3.from spark dataframe vectorUDT to generate 

  here  need to import three package,some complex convert to generate,  I  think  maybe could from ndarray return regenerate spark dataframe  to write like csv or parquet file 

4.also from Operand[T] tensor  or  tfrecord Dataset
   here  I only see  from   ndArray to generate tensor ,and from  Dataset to generate ndArray
...