Memory consomption

[PhidarkMM]

Hi all,

Recently we were asked to modify the limits of certain shared spaces in order to transmit large files > 10GB (databases, debug reports, etc...).

It seems that linshare has a working limit equal to the size of its java memory (for us JAVA_OPTS=...-Xms512m -Xmx4096m)

How is memory managed in linshare? Should the java memory reservation size be that of the largest loads ? What if there are several large uploads simultaneously?

Thanking you, Best regards,

[wboudiche]

Hi What is the type of storage you are using ?

[PhidarkMM]

Hi,

We are using VmWare and HP 3Par storage.

Regards,

[PhidarkMM]

All of our storage options:

#### Global storage options #### # available storage mode : # filesystem : storing documents on file system for tests # swift-keystone : storing documents into swift linshare.documents.storage.mode=filesystem linshare.documents.storage.bucket=e0531829-8a75-49f8-bb30-4539574d66c7 # Experimental : use this key with caution ! linshare.documents.storage.deduplication=true

# Temporary directory : local work directory to encrypt/decrypt data linshare.encipherment.tmp.dir=/linshare/tmp

#### File system storage options - default backend#### linshare.documents.storage.filesystem.directory=/linshare/filesystemstorage

######## Extended storage options ########

#### Object storage options #### # deprecated property, if set user.name and user.domain will be ignored. linshare.documents.storage.identity= # NB : For OpenStack Swift, format is different # linshare.documents.storage.identity=tenant_name:user_name linshare.documents.storage.user.domain= linshare.documents.storage.user.name= linshare.documents.storage.credential= # LinShare supports keystone2 and keystone3 for swift. linshare.documents.storage.keystone.version=2 # Project name is required for keystone 3 (useless for keystone s2) linshare.documents.storage.project.name= # keystone endpoint, ex : # - https://auth.cloud.ovh.net/v2.0/ # - https://auth.cloud.ovh.net/v3 linshare.documents.storage.endpoint= # required if you are using Swift with region (openstack-swift) linshare.documents.storage.regionId= linshare.documents.storage.multipartupload=true

#### Swift storage options #### linshare.documents.storage.swift.identity=tenant_name:user_name linshare.documents.storage.swift.credential=password linshare.documents.storage.swift.endpoint=http://127.0.0.1:5000/v2.0

#### GridFS storage options #### linshare.mongo.host=127.0.0.1 linshare.mongo.port=27017 linshare.mongo.user=linshare linshare.mongo.password=xxx linshare.mongo.database=linshare linshare.mongo.connect.timeout=30000 linshare.mongo.socket.timeout=30000

linshare.mongo.gridfs.smallfiles.host=127.0.0.1 linshare.mongo.gridfs.smallfiles.port=27017 linshare.mongo.gridfs.smallfiles.user=linshare linshare.mongo.gridfs.smallfiles.password=xxx linshare.mongo.gridfs.smallfiles.database=linshare-files

#### Write concern # MAJORITY: waits on a majority of servers for the write operation. # JOURNALED: Write operations wait for the server to group commit to the journal file on disk. #### Write concern # MAJORITY: waits on a majority of servers for the write operation. # JOURNALED: Write operations wait for the server to group commit to the journal file on disk. # ACKNOWLEDGED: Write operations that use this write concern will wait for acknowledgement, # using the default write concern configured on the server. linshare.mongo.write.concern=MAJORITY

#### connection for data # replicaset: host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]] linshare.mongo.data.replicaset=127.0.0.1:27017 linshare.mongo.data.database=linshare # linshare.mongo.data.credentials=[user:password[@database]] linshare.mongo.data.credentials=linshare:xxx@admin

#### connection for small files # Using MongoDb to store very small files (thumbnails, mail attachments, ...) linshare.mongo.smallfiles.replicaset=127.0.0.1:27017 linshare.mongo.smallfiles.database=linshare-files linshare.mongo.smallfiles.credentials=linshare:xxx@admin

#### connection for big files. (dev mode) # Store all files in MongoDB GridFS. Not recommended. #linshare.mongo.bigfiles.replicaset=127.0.0.1:27017 #linshare.mongo.bigfiles.database=linshare-bigfiles #linshare.mongo.bigfiles.credentials=

#### Extended GridFS storage options #### # Store all files in MongoDB GridFS. Not recommended. #linshare.mongo.gridfs.bigfiles.host=127.0.0.1 #linshare.mongo.gridfs.bigfiles.port=27017 #linshare.mongo.gridfs.bigfiles.user=linshare #linshare.mongo.gridfs.bigfiles.password=linshare #linshare.mongo.gridfs.bigfiles.database=linshare-bigfiles

######## Storage options - end ########

[wboudiche]

Hi, LinShare use multipart upload and streaming so you don't have to have the available memory equal to file size to upload. But you need to tune and test to have the best configuration.

[PhidarkMM]

Hello,

Thank you @wboudiche

After some tests I was able to download a 14GB file without java memory outage (jcloud batch upload)

ls -ltr /linshare/filesystemstorage/e0531829-8a75-49f8-bb30-4539574d66c7/

... -rw-r----- 1 tomcat tomcat 14765509297 Nov 8 11:13 d849f026-44d2-423b-898e-3cba181b9273

Local filesystem storage configuration: ######## Storage options ########

#### Global storage options #### # available storage mode : # filesystem : storing documents on file system for tests # swift-keystone : storing documents into swift linshare.documents.storage.mode=filesystem linshare.documents.storage.bucket=e0531829-8a75-49f8-bb30-4539574d66c7 # Experimental : use this key with caution ! linshare.documents.storage.deduplication=true

# Temporary directory : local work directory to encrypt/decrypt data linshare.encipherment.tmp.dir=/linshare/tmp

#### File system storage options - default backend#### linshare.documents.storage.filesystem.directory=/linshare/filesystemstorage

######## Extended storage options ########

#### Object storage options #### # deprecated property, if set user.name and user.domain will be ignored. #linshare.documents.storage.identity= # NB : For OpenStack Swift, format is different # linshare.documents.storage.identity=tenant_name:user_name #linshare.documents.storage.user.domain= #linshare.documents.storage.user.name= #linshare.documents.storage.credential= # LinShare supports keystone2 and keystone3 for swift. #linshare.documents.storage.keystone.version=2 # Project name is required for keystone 3 (useless for keystone s2) #linshare.documents.storage.project.name= # keystone endpoint, ex : # - https://auth.cloud.ovh.net/v2.0/ # - https://auth.cloud.ovh.net/v3 #linshare.documents.storage.endpoint= # required if you are using Swift with region (openstack-swift) #linshare.documents.storage.regionId= #linshare.documents.storage.multipartupload=true

#### Swift storage options #### #linshare.documents.storage.swift.identity=tenant_name:user_name #linshare.documents.storage.swift.credential=password #linshare.documents.storage.swift.endpoint=http://127.0.0.1:5000/v2.0\

#### GridFS storage options #### linshare.mongo.host=127.0.0.1 linshare.mongo.port=27017 linshare.mongo.user=linshare linshare.mongo.password=xxx linshare.mongo.database=linshare linshare.mongo.connect.timeout=30000 linshare.mongo.socket.timeout=30000

linshare.mongo.gridfs.smallfiles.host=127.0.0.1 linshare.mongo.gridfs.smallfiles.port=27017 linshare.mongo.gridfs.smallfiles.user=linshare linshare.mongo.gridfs.smallfiles.password=xxx linshare.mongo.gridfs.smallfiles.database=linshare-files

#### Write concern # MAJORITY: waits on a majority of servers for the write operation. # JOURNALED: Write operations wait for the server to group commit to the journal file on disk. #### Write concern # MAJORITY: waits on a majority of servers for the write operation. # JOURNALED: Write operations wait for the server to group commit to the journal file on disk. # ACKNOWLEDGED: Write operations that use this write concern will wait for acknowledgement, # using the default write concern configured on the server. linshare.mongo.write.concern=MAJORITY

#### connection for data # replicaset: host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]] linshare.mongo.data.replicaset=127.0.0.1:27017 linshare.mongo.data.database=linshare # linshare.mongo.data.credentials=[user:password[@database]] linshare.mongo.data.credentials=linshare:xxx@admin

#### connection for small files # Using MongoDb to store very small files (thumbnails, mail attachments, ...) linshare.mongo.smallfiles.replicaset=127.0.0.1:27017 linshare.mongo.smallfiles.database=linshare-files linshare.mongo.smallfiles.credentials=linshare:xxx@admin

#### connection for big files. (dev mode) # Store all files in MongoDB GridFS. Not recommended. linshare.mongo.bigfiles.replicaset=127.0.0.1:27017 linshare.mongo.bigfiles.database=linshare-bigfiles linshare.mongo.bigfiles.credentials=linshare:xxx@admin

#### Extended GridFS storage options #### # Store all files in MongoDB GridFS. Not recommended. linshare.mongo.gridfs.bigfiles.host=127.0.0.1 linshare.mongo.gridfs.bigfiles.port=27017 linshare.mongo.gridfs.bigfiles.user=linshare linshare.mongo.gridfs.bigfiles.password=xxx linshare.mongo.gridfs.bigfiles.database=linshare-bigfiles

######## Storage options - end ########