search

maccmu / macposts

MIT License
9 stars 3 forks source link

Switch to `float` from `double` for data? #66

Open ghost opened 2 months ago

ghost commented 2 months ago

Right now we are using double precision floating point numbers (namely double) in our code base to store data. While that gives more accurate results and is less vulnerable to rounding errors, it also increases the memory usage of the application, compared to float. So I am considering to use the single precision type float more in our code base.

For my use cases, I do not need such high precision in most places (but I am still investigating). I wonder if anyone has some cases that double is actually required?

psychogeekir commented 2 months ago

Float is accurate enough for my cases.

Qiling Zou

From: kunhtkun @.> Sent: Tuesday, June 18, 2024 5:28:41 AM To: maccmu/macposts @.> Cc: Subscribed @.***> Subject: [maccmu/macposts] Switch to float from double for internal data storage? (Issue #66)

Right now we are using double precision floating point numbers (namely double) in our code base to store data. While that gives more accurate results and is less vulnerable to rounding errors, it also increases the memory usage of the application, compared to float. So I am considering to use the single precision type float more in our code base.

For my use cases, I do not need such high precision in most places (but I am still investigating). I wonder if anyone has some cases that double is actually required?

— Reply to this email directly, view it on GitHubhttps://github.com/maccmu/macposts/issues/66, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABXEA7N4IBQD7NEAPQF377LZH5IITAVCNFSM6AAAAABJOYTARCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM2TQMRZGMYDKOI. You are receiving this because you are subscribed to this thread.Message ID: @.***>

ghost commented 2 months ago

Thanks for the input! I am test driving a float build and will report back after I get some results.

ghost commented 2 months ago

I did a quick and dirty swap of double to float, and ran it on the Sioux Falls network. The results were not as impressive as I expected though. Below are two graphs showing the memory usage overtime (x-axis is time, and y-axis is heap usage), produced by valgrind --tool=massif and ms_print.

Using float:

    MB
8.087^                                         #                              
     |                                       ::#:::                           
     |                                   @:::::#:::@:::         :::@:::::@::: 
     |                                :@:@: :::#:::@:::@::::@::::::@:::::@::: 
     |                              :::@:@: :::#:::@:::@::::@::::::@:::::@::: 
     |                              :::@:@: :::#:::@:::@::::@::::::@:::::@::: 
     |                            :::::@:@: :::#:::@:::@::::@::::::@:::::@::: 
     |                           @: :::@:@: :::#:::@:::@::::@::::::@:::::@::: 
     |                          :@: :::@:@: :::#:::@:::@::::@::::::@:::::@::: 
     |                        :::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
     |                        :::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
     |                       @:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
     |                    @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
     |               :::@:@::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
     |            @@::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
     |           :@ ::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
     |         :::@ ::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
     |      @:::::@ ::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
     |    ::@:::::@ ::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
     | @::::@:::::@ ::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@::::
   0 +----------------------------------------------------------------------->Mi
     0                                                                   864.8

Using double:

    MB
8.631^                                         #                              
     |                                       ::#::::                       :: 
     |                                     ::: #: ::::::::::::::::::::::::::  
     |                                  :::::: #: ::: :: :: ::: ::: ::: :: :  
     |                                @@:: ::: #: ::: :: :: ::: ::: ::: :: :  
     |                              ::@ :: ::: #: ::: :: :: ::: ::: ::: :: :  
     |                              : @ :: ::: #: ::: :: :: ::: ::: ::: :: :  
     |                            ::: @ :: ::: #: ::: :: :: ::: ::: ::: :: :  
     |                           @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |                        @@@@: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |                        @@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |                        @@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |                       @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |                   ::::@@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |              @@:::: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |            ::@ : :: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |           :::@ : :: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |         :::::@ : :: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |     @:::: :::@ : :: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
     |  :::@: :: :::@ : :: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : :
   0 +----------------------------------------------------------------------->Mi
     0                                                                   859.9

It is only ~6% reduction on memory usage. Perhaps the network is too small?

The script:

import macposts
import macposts.utils

with macposts.utils.silence():
    dta = macposts.Dta.from_files("siouxfalls")
    dta.register_links()
    dta.install_cc()
    dta.run_whole()

Data files can be downloaded here: https://babyam.andrew.cmu.edu/files/macposts/data/siouxfalls.zip

psychogeekir commented 2 months ago

Thanks for the efforts. I agree that the improvement looks somewhat minor for this network. But it may still be significant for large networks.

On Thu, Jun 20, 2024 at 8:17 AM kunhtkun @.***> wrote:

I did a quick and dirty swap of double to float, and ran it on the Sioux Falls network. The results were not as impressive as I expected though. Below are two graphs showing the memory usage overtime (x-axis is time, and y-axis is heap usage), produced by valgrind --tool=massif and ms_print.

Using float:

MB

8.087^ # | ::#::: | @:::::#:::@::: :::@:::::@::: | :@:@: :::#:::@:::@::::@::::::@:::::@::: | :::@:@: :::#:::@:::@::::@::::::@:::::@::: | :::@:@: :::#:::@:::@::::@::::::@:::::@::: | :::::@:@: :::#:::@:::@::::@::::::@:::::@::: | @: :::@:@: :::#:::@:::@::::@::::::@:::::@::: | :@: :::@:@: :::#:::@:::@::::@::::::@:::::@::: | :::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: | :::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: | @:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: | @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: | :::@:@::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: | @@::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: | :@ ::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: | :::@ ::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: | @:::::@ ::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: | ::@:::::@ ::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: | @::::@:::::@ ::: @ @::@:::@: :::@:@: :::#:::@:::@::::@::::::@:::::@:::: 0 +----------------------------------------------------------------------->Mi 0 864.8

Using double:

MB

8.631^ # | ::#:::: :: | ::: #: :::::::::::::::::::::::::: | :::::: #: ::: :: :: ::: ::: ::: :: : | @@:: ::: #: ::: :: :: ::: ::: ::: :: : | ::@ :: ::: #: ::: :: :: ::: ::: ::: :: : | : @ :: ::: #: ::: :: :: ::: ::: ::: :: : | ::: @ :: ::: #: ::: :: :: ::: ::: ::: :: : | @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | @@@@: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | @@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | @@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | ::::@@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | @@:::: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | ::@ : :: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | :::@ : :: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | :::::@ : :: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | @:::: :::@ : :: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : | :::@: :: :::@ : :: : @@@ @: : @ :: ::: #: ::: :: :: ::: ::: ::: :: : : 0 +----------------------------------------------------------------------->Mi 0 859.9

It is only ~6% reduction on memory usage. Perhaps the network is too small?

The script:

import macpostsimport macposts.utils with macposts.utils.silence(): dta = macposts.Dta.from_files("siouxfalls") dta.register_links() dta.install_cc() dta.run_whole()

Data files can be downloaded here: https://babyam.andrew.cmu.edu/files/macposts/data/siouxfalls.zip

— Reply to this email directly, view it on GitHub https://github.com/maccmu/macposts/issues/66#issuecomment-2179601207, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXEA7J34F2SKFJFHAT7SZTZIINRHAVCNFSM6AAAAABJOYTARCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZZGYYDCMRQG4 . You are receiving this because you commented.Message ID: @.***>