search

epics-base / pvaPy

pvaPy provides Python bindings for EPICS pvAccess
https://epics.anl.gov/extensions/pvaPy/production/index.html
Other
36 stars 22 forks source link

maximum array size limitation #93

Closed swelborn closed 2 months ago

swelborn commented 2 months ago

Hello. When you run the script below, there is a maximum size limitation. I am wondering what this is coming from, as the error does give us anything (it is blank). I was under the impression that there was no limitation in size for PVA objects, as opposed to CA objects which require us to set an environment variable (cannot remember exactly, something like EPICS_CA_MAX_ARRAY_SIZE).

import logging
import time

import numpy as np
import pvaccess as pva
from pvapy.utility.adImageUtility import AdImageUtility

logging.basicConfig(
    level=logging.DEBUG, format="%(asctime)s - %(levelname)s - %(message)s"
)
logger = logging.getLogger("util.AdImageUtility3D")

class AdImageUtility3D(AdImageUtility):

    @classmethod
    def generateNtNdArray3D(
        cls,
        imageId: int,
        imageData: np.ndarray,
        extraFieldsPvObject: pva.PvObject | None = None,
    ):
        """Generate NTNDA for a 3D image."""
        if not extraFieldsPvObject:
            ntNdArray = pva.NtNdArray()
        else:
            ntNdArray = pva.NtNdArray(extraFieldsPvObject.getStructureDict())

        if imageData.ndim != 3:
            raise pva.InvalidArgument("Input data must be a 3D numpy array.")

        dataFieldKey = cls.NTNDA_DATA_FIELD_KEY_MAP.get(imageData.dtype)
        logger.debug("Raveling image data")
        data = imageData.ravel()

        nz, ny, nx = imageData.shape
        size = nx * ny * nz * data.itemsize
        logger.debug(f"Image size: {nx}x{ny}x{nz}, data size: {size}")
        ntNdArray["compressedSize"] = size
        ntNdArray["uncompressedSize"] = size

        ntNdArray["uniqueId"] = int(imageId)
        dims = [
            pva.PvDimension(nz, 0, nz, 1, False),
            pva.PvDimension(ny, 0, ny, 1, False),
            pva.PvDimension(nx, 0, nx, 1, False),
        ]
        logger.debug(f"Setting dimensions: {dims}")
        ntNdArray["dimension"] = dims
        ts = pva.PvTimeStamp(time.time())
        ntNdArray["timeStamp"] = ts
        ntNdArray["dataTimeStamp"] = ts
        ntNdArray["descriptor"] = "3D Image generated by PvaPy"

        logger.debug(f"Setting data field {dataFieldKey}")
        ntNdArray["value"] = {dataFieldKey: data}
        logger.debug("Setting extra fields")
        if extraFieldsPvObject:
            ntNdArray.set(extraFieldsPvObject)
        logger.debug("Returning NTNDArray")
        return ntNdArray

def test_maximum_size():
    min_size = 1025
    max_size = 1035
    step = 1

    while min_size <= max_size:
        size = (min_size, 2500, 2500)
        try:
            logger.info(f"Testing size: {size[0]}x{size[1]}x{size[2]}")
            image_data = np.zeros(size, dtype=np.uint8)
            AdImageUtility3D.generateNtNdArray3D(1, image_data)
            logger.info(
                f"Successfully generated NTNDArray for size: {size[0]}x{size[1]}x{size[2]}"
            )
            min_size += step
        except Exception as e:
            logger.error(
                f"Failed to generate NTNDArray for size: {size[0]}x{size[1]}x{size[2]}"
            )
            logger.error(f"Error: {str(e)}")
            max_size = size[0] - 1
            step = max(1, step // 2)

    logger.info(f"Maximum successful size: {min_size - step}x2500x2500")

if __name__ == "__main__":
    test_maximum_size()

Here is the output:

python scripts/test_sizes.py 
2024-07-17 15:14:50,108 - INFO - Testing size: 1025x2500x2500
2024-07-17 15:14:50,108 - DEBUG - Raveling image data
2024-07-17 15:14:50,108 - DEBUG - Image size: 2500x2500x1025, data size: 6406250000
2024-07-17 15:14:50,108 - DEBUG - Setting dimensions: [<pvaccess.pvaccess.PvDimension object at 0x71b73a8c6c50>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6e30>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6f70>]
2024-07-17 15:14:50,108 - DEBUG - Setting data field ubyteValue
2024-07-17 15:14:51,675 - DEBUG - Setting extra fields
2024-07-17 15:14:51,676 - DEBUG - Returning NTNDArray
2024-07-17 15:14:51,842 - INFO - Successfully generated NTNDArray for size: 1025x2500x2500
2024-07-17 15:14:51,842 - INFO - Testing size: 1026x2500x2500
2024-07-17 15:14:51,851 - DEBUG - Raveling image data
2024-07-17 15:14:51,851 - DEBUG - Image size: 2500x2500x1026, data size: 6412500000
2024-07-17 15:14:51,851 - DEBUG - Setting dimensions: [<pvaccess.pvaccess.PvDimension object at 0x71b73b073b50>, <pvaccess.pvaccess.PvDimension object at 0x71b73b03c310>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6ed0>]
2024-07-17 15:14:51,851 - DEBUG - Setting data field ubyteValue
2024-07-17 15:14:53,407 - DEBUG - Setting extra fields
2024-07-17 15:14:53,407 - DEBUG - Returning NTNDArray
2024-07-17 15:14:53,579 - INFO - Successfully generated NTNDArray for size: 1026x2500x2500
2024-07-17 15:14:53,579 - INFO - Testing size: 1027x2500x2500
2024-07-17 15:14:53,588 - DEBUG - Raveling image data
2024-07-17 15:14:53,588 - DEBUG - Image size: 2500x2500x1027, data size: 6418750000
2024-07-17 15:14:53,588 - DEBUG - Setting dimensions: [<pvaccess.pvaccess.PvDimension object at 0x71b78bead350>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6cf0>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6ed0>]
2024-07-17 15:14:53,589 - DEBUG - Setting data field ubyteValue
2024-07-17 15:14:55,141 - DEBUG - Setting extra fields
2024-07-17 15:14:55,141 - DEBUG - Returning NTNDArray
2024-07-17 15:14:55,309 - INFO - Successfully generated NTNDArray for size: 1027x2500x2500
2024-07-17 15:14:55,309 - INFO - Testing size: 1028x2500x2500
2024-07-17 15:14:55,318 - DEBUG - Raveling image data
2024-07-17 15:14:55,318 - DEBUG - Image size: 2500x2500x1028, data size: 6425000000
2024-07-17 15:14:55,318 - DEBUG - Setting dimensions: [<pvaccess.pvaccess.PvDimension object at 0x71b73a8c6f70>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6de0>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6ed0>]
2024-07-17 15:14:55,318 - DEBUG - Setting data field ubyteValue
2024-07-17 15:14:56,871 - DEBUG - Setting extra fields
2024-07-17 15:14:56,871 - DEBUG - Returning NTNDArray
2024-07-17 15:14:57,042 - INFO - Successfully generated NTNDArray for size: 1028x2500x2500
2024-07-17 15:14:57,042 - INFO - Testing size: 1029x2500x2500
2024-07-17 15:14:57,050 - DEBUG - Raveling image data
2024-07-17 15:14:57,050 - DEBUG - Image size: 2500x2500x1029, data size: 6431250000
2024-07-17 15:14:57,050 - DEBUG - Setting dimensions: [<pvaccess.pvaccess.PvDimension object at 0x71b788d39490>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6de0>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6f70>]
2024-07-17 15:14:57,051 - DEBUG - Setting data field ubyteValue
2024-07-17 15:14:58,689 - DEBUG - Setting extra fields
2024-07-17 15:14:58,689 - DEBUG - Returning NTNDArray
2024-07-17 15:14:58,871 - INFO - Successfully generated NTNDArray for size: 1029x2500x2500
2024-07-17 15:14:58,871 - INFO - Testing size: 1030x2500x2500
2024-07-17 15:14:58,880 - DEBUG - Raveling image data
2024-07-17 15:14:58,880 - DEBUG - Image size: 2500x2500x1030, data size: 6437500000
2024-07-17 15:14:58,880 - DEBUG - Setting dimensions: [<pvaccess.pvaccess.PvDimension object at 0x71b73a8c6de0>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c7290>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6cf0>]
2024-07-17 15:14:58,880 - DEBUG - Setting data field ubyteValue
2024-07-17 15:15:00,569 - DEBUG - Setting extra fields
2024-07-17 15:15:00,569 - DEBUG - Returning NTNDArray
2024-07-17 15:15:00,741 - INFO - Successfully generated NTNDArray for size: 1030x2500x2500
2024-07-17 15:15:00,741 - INFO - Testing size: 1031x2500x2500
2024-07-17 15:15:00,750 - DEBUG - Raveling image data
2024-07-17 15:15:00,750 - DEBUG - Image size: 2500x2500x1031, data size: 6443750000
2024-07-17 15:15:00,750 - DEBUG - Setting dimensions: [<pvaccess.pvaccess.PvDimension object at 0x71b78bffb740>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c7290>, <pvaccess.pvaccess.PvDimension object at 0x71b73a8c6e30>]
2024-07-17 15:15:00,750 - DEBUG - Setting data field ubyteValue
2024-07-17 15:15:00,750 - ERROR - Failed to generate NTNDArray for size: 1031x2500x2500
2024-07-17 15:15:00,750 - ERROR - Error: 
2024-07-17 15:15:00,750 - INFO - Maximum successful size: 1030x2500x2500
sveseli commented 2 months ago

Thanks for reporting this, the problem will be resolved in the next release: see https://github.com/epics-base/pvaPy/commit/4a66e0e97639733980520071d76f2cf2d0fe6287

sveseli commented 2 months ago

This should be now resolved as of release 5.4.1.