Too many files open

[ghansham]

We have a frame camera that covers a large area on earth by stitching frames. We found that elev classes keeps opening files and at times it crosses the system limit of 1024. Do we have a way to restrict the number of files opened by ossim Elevation classes. We are sure because we checked using lsof for the running process. We are running on rhel 7.X.

[omarossim]

Hello:

There should be an ossim preferences file referenced via an environment variable that has entries sort of like this for elevation:

elevation_manager.elevation_source1.type: dted_directory elevation_manager.elevation_source1.connection_string: /Volumes/DataDrive/data/elevation/dted/level2 elevation_manager.elevation_source1.min_open_cells: 5 elevation_manager.elevation_source1.max_open_cells: 25 elevation_manager.elevation_source1.enabled: true elevation_manager.elevation_source1.memory_map_cells: false

there is a min_open_cells and a max that restrict for that database. If it’s a low res 1K cell then you can enable memory mapping and it will pull it into memory when loading.

There is a template that is annotated under

share/ossim/templates/ossim_preferences_template

Take care

Garrett

On Apr 5, 2019, at 11:30 PM, ghansham notifications@github.com wrote:

We have a frame camera that covers a large area on earth by stitching frames. We found that elev classes keeps opening files and at times it crosses the system limit of 1024. Do we have a way to restrict the number of files opened by ossim Elevation classes. We are sure because we checked using lsof for the running process. We are running on rhel 7.X.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ossimlabs/ossim/issues/218, or mute the thread https://github.com/notifications/unsubscribe-auth/AGpvyoGxtMHpcjOvLPyvHRucqr6RKzpXks5veBTFgaJpZM4cf_Z1.

[ghansham]

Thanks. I will try the suggested solution.

[ghansham]

This worked. Actually issue was coming because ossim elevation manager threads were enabled. By default it was using 32 threads. Another observation is that same file was opened by multiple ossim elevation database instances. Is that fine? Another suggestion was while loading the default paths for elevation, it should check existence of those paths before instantiating the relevant classes.

[omarossim]

Hello:

Yes, just looked at the implementation. It uses a round robin so each thread has the potential to open the same elevation data base. The actual sources are not shared so they will get replicated.

Take care

Garrett

On Apr 13, 2019, at 11:23 AM, ghansham notifications@github.com wrote:

This worked. Actually issue was coming because ossim elevation manager threads were enabled. By default it was using 32 threads. Another observation is that same file was opened by multiple ossim elevation database instances. Is that fine? Another suggestion was while loading the default paths for elevation, it should check existence of those paths before instantiating the relevant classes.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ossimlabs/ossim/issues/218#issuecomment-482821282, or mute the thread https://github.com/notifications/unsubscribe-auth/AGpvynrVtBk_kZqtb32rBze9Vpzjdb0zks5vgfaHgaJpZM4cf_Z1.

[omarossim]

Hello:

Looking at the check for existance and see

Take care

Garrett

On Apr 13, 2019, at 11:23 AM, ghansham notifications@github.com wrote:

This worked. Actually issue was coming because ossim elevation manager threads were enabled. By default it was using 32 threads. Another observation is that same file was opened by multiple ossim elevation database instances. Is that fine? Another suggestion was while loading the default paths for elevation, it should check existence of those paths before instantiating the relevant classes.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ossimlabs/ossim/issues/218#issuecomment-482821282, or mute the thread https://github.com/notifications/unsubscribe-auth/AGpvynrVtBk_kZqtb32rBze9Vpzjdb0zks5vgfaHgaJpZM4cf_Z1.

[omarossim]

Hello:

For the check if exists we currenlty check to see if the database has one sell in it before adding it to the manager. Is that what you are asking?

Take care

Garrett

On Apr 13, 2019, at 11:23 AM, ghansham notifications@github.com wrote:

This worked. Actually issue was coming because ossim elevation manager threads were enabled. By default it was using 32 threads. Another observation is that same file was opened by multiple ossim elevation database instances. Is that fine? Another suggestion was while loading the default paths for elevation, it should check existence of those paths before instantiating the relevant classes.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ossimlabs/ossim/issues/218#issuecomment-482821282, or mute the thread https://github.com/notifications/unsubscribe-auth/AGpvynrVtBk_kZqtb32rBze9Vpzjdb0zks5vgfaHgaJpZM4cf_Z1.

[ghansham]

In the loadStandardElevationPaths method in ElevManager class, the loadElevationPath method is called for userPath and installPath without checking for their existence. It checks for existence after calling open on the path. That open is even is also not required. See line number 214 in loadElevationPath. The path existence should be checked within loadStandardElevationPaths. That may do away with with open function call which is not required if path does not exist.

[ghansham]

Regarding sharing of elevation managers, if the cells are mapped to memory, they can be shared, that can reduce the memory foot print. Am I correct?

[ghansham]

I think loadElevationPath should have existence check before calling open because this method might be getting called from multiple places other than loadStandardElevationPaths.

Regards Ghansham

[gpotts]

Hello:

Let me see about the elevation path. We moved checks into the factories instead because the ossimFilename is actually overloaded based on protocol type. Some things will not work with exists checks. The elevation database might not be file based and so a straight file exists will prematurely exit and never load a database with a different protocol for loading elevation.

Will have to think about this one for a few and see. As it sits now, this is typically called only during initialization of ossim. After initialized this is no longer called.

Tale care

Garrett

From: ghansham notifications@github.com Reply-To: ossimlabs/ossim reply@reply.github.com Date: Sunday, April 14, 2019 at 9:17 PM To: ossimlabs/ossim ossim@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [ossimlabs/ossim] Too many files open (#218)

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I think loadElevationPath should have existence check before calling open because this method might be getting called from multiple places other than loadStandardElevationPaths.

Regards Ghansham

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/ossimlabs/ossim/issues/218#issuecomment-483077076, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AARP3P4NBRVXGUN3BF2LQU3PQPJRPANCNFSM4HD76Z2Q.

This electronic communication and any attachments may contain confidential and proprietary information of Radiant Solutions, Inc. If you are not the intended recipient, or an agent or employee responsible for delivering this communication to the intended recipient, or if you have received this communication in error, please do not print, copy, retransmit, disseminate or otherwise use the information. Please indicate to the sender that you have received this communication in error, and delete the copy you received.

Radiant Solutions reserves the right to monitor any electronic communication sent or received by its employees, agents or representatives.

[ghansham]

Ok. I agree. Yeah its only during initialization. Idea too was to speed up the startup time only.