search

qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
https://qgis.org
GNU General Public License v2.0
10.65k stars 3.01k forks source link

macOS qgis.app diet #59038

Open jaredmauch opened 1 month ago

jaredmauch commented 1 month ago

Feature description

When installing QGIS the file on macOS takes awhile to copy, even with fast storage (eg: nvme) - A fair amount of this time is copying small files which result in many i/o operations to the filesystem.

I would like to suggest that the data be moved either into a blob, eg: https://www.sqlite.org/fasterthanfs.html or that some of the files might be removed.

For example, I believe it may not be necessary to include many of these files, or they could be archived in a way that it doesn't take as much disk i/o to copy the app from the disk image to the storage.

open Contents/Resources/python/site-packages/cmake/data/CMake.app/Contents/doc/cmake/html/release/
3.0.html    3.10.html   3.12.html   3.14.html   3.16.html   3.18.html   3.2.html    3.21.html   3.4.html    3.6.html    3.8.html    index.html  
3.1.html    3.11.html   3.13.html   3.15.html   3.17.html   3.19.html   3.20.html   3.3.html    3.5.html    3.7.html    3.9.html

Additional context

I monitor the time it takes to copy by running iostat -d 1 in an alternative window. Extract from the iostat man page for macos:

           The standard iostat device display shows the following statistics:

           KB/t    kilobytes per transfer
           tps     transfers per second
           MB/s    megabytes per second

This shows a high number of tps (open/read/close) of files and a reduced I/O rate of the attached storage during the copy operation from the disk image (disk2) to the internal storage (disk0) 

  Physical Drive:
  Device Name:  Samsung SSD 970 EVO Plus 2TB
  Media Name:   AppleAPFSMedia
  Medium Type:  SSD
  Protocol: PCI-Express
  Internal: Yes
              disk0               disk2 
    KB/t  tps  MB/s     KB/t  tps  MB/s 
  235.40   20  4.58   513.88   17  8.50 
  432.57   28 11.82   616.00   13  7.82 
  504.12   33 16.18  1663.60   10 16.18 
  367.00    8  2.85  1403.56    9 12.28 
  481.29   34 15.95  1418.40   10 13.83 
  473.29   34 15.63  1752.50    8 13.62 
  487.65   23 10.93  1657.50    8 12.93 
  459.33   24 10.76  1633.00    8 12.75 
  370.31   45 16.22   156.29   28  4.26 
    9.60   15  0.14     4.80   30  0.14 
   12.21  190  2.27    53.44   25  1.30 
    7.53  323  2.38    25.56   41  1.02 
   11.18  249  2.72     9.28  262  2.37 
   16.37  394  6.30    13.60  438  5.82 
   15.42  297  4.48    12.76  329  4.10 
   29.29  250  7.16    27.93  249  6.80 
    9.97  661  6.44     7.35  845  6.07 
   45.95  119  5.33   248.26   62 14.99 
  275.12  144 38.75  1721.05   19 31.76 
  253.73  153 37.84   853.37   38 31.61 
              disk0               disk2 
    KB/t  tps  MB/s     KB/t  tps  MB/s 
   71.29  140  9.73   102.04   92  9.16 
   17.32  334  5.65    14.01  394  5.39 
   20.90   98  2.00   368.84   38 13.68 
  176.99  101 17.45  1809.00   12 21.19 
  266.29  140 36.34  1586.46   13 20.10 
  190.56  108 20.03  1846.18   11 19.76 
   10.10 1151 11.35     9.73 1152 10.94 
    4.31  799  3.36     4.21  760  3.12 
   10.51 1131 11.61    13.31  497  6.45 
   23.91  401  9.36     8.22  674  5.41 
   12.59  656  8.07     6.22 1209  7.35 
   12.00  121  1.41     4.06  140  0.55 
   33.02  429 13.84     4.21  133  0.55 
    8.90  283  2.46     4.35  253  1.07 
    9.26  341  3.09     4.15  577  2.34 
   93.04  149 13.58    94.07  146 13.45 
  199.93  110 21.39  1012.89   18 17.73 
  260.36  101 25.66  1520.00   15 22.25 
   12.89  592  7.45    10.53  603  6.20 
    5.58  791  4.32     4.22  775  3.19 
              disk0               disk2 
    KB/t  tps  MB/s     KB/t  tps  MB/s 
    7.04  836  5.75     4.99  759  3.70 
   45.34  119  5.26    19.06   81  1.50 
   54.45  270 14.35    22.14  324  7.00 
   28.36  509 14.09    12.82  604  7.57 
   25.33  380  9.40    17.29  477  8.05 
   31.62  271  8.38    15.27  312  4.66 
   23.25  561 12.75    11.82  661  7.63 
   25.76  209  5.25    91.34  200 17.80 
   79.56   72  5.58  2048.00   14 27.93 
  265.24  293 75.81   207.62  147 29.77 
   35.65  162  5.63    22.06  238  5.13 
   19.79 1172 22.66    40.00  170  6.65 
   28.38  181  5.01    20.86  244  4.96 
   14.65  444  6.35     9.89  657  6.34 
   13.23  849 10.97     8.55 1313 10.96 
   14.18  336  4.66     9.30  513  4.66 
  102.61   72  7.20   111.08   65  7.04 
   25.52  446 11.11    18.45  597 10.76 
   18.79 1778 32.64    79.12  222 17.18 
   32.93   56  1.79  2048.00   12 23.87 
              disk0               disk2 
    KB/t  tps  MB/s     KB/t  tps  MB/s 
  351.77  176 60.44  2048.00   12 23.99 
  151.88  172 25.44   187.44   94 17.16 
   48.45  160  7.59    44.52  146  6.33 
   60.11  147  8.66   117.60   60  6.87 
   14.36  415  5.82    53.31   77  3.99 
   24.85  522 12.68    30.42  389 11.56 
   11.35  825  9.14     8.84  960  8.28 
   11.48  718  8.05     8.65  825  6.96 
   13.91  616  8.37    10.60  763  7.90 
   18.17  554  9.84     9.76  575  5.48 
   13.23  546  7.06     9.72  659  6.25 
   22.07  405  8.73    20.98  400  8.19 
   25.19  428 10.54    22.03  446  9.60 
    9.29 1805 16.37    16.52  583  9.40 
    8.23  771  6.19     7.07  746  5.15 
   25.36  403  9.97    20.58  367  7.38 
   22.65  380  8.41    19.23  428  8.04 
  119.16  118 13.77   213.54   65 13.48 
   79.45  146 11.30    94.63  120 11.06 
    5.04  938  4.62     4.71  907  4.17 
              disk0               disk2 
    KB/t  tps  MB/s     KB/t  tps  MB/s 
   28.65  152  4.26    79.49   47  3.63 
    7.17  819  5.73     7.84  179  1.37 
   24.57  887 21.28    19.98  596 11.63 
   77.59  116  8.75    51.67  123  6.18 
   25.84  457 11.52    27.97  311  8.51 
   70.11  198 13.58    51.00  238 11.86 
   65.36  158 10.10    47.75  173  8.07 
   27.80  578 15.69    25.25  532 13.12 
   17.97  748 13.12    12.33  464  5.58 
   34.61  270  9.12    19.94  367  7.15 
   33.41  203  6.63    21.25  256  5.31 
   22.02  542 11.65    20.67  453  9.15 
   24.05  429 10.08    14.74  599  8.63 
   30.59  361 10.77    33.76  261  8.60 
   36.17  324 11.43    25.31  398  9.85 
   24.80  466 11.27    13.39  727  9.50 
   32.38  411 13.01    42.97  245 10.27 
   17.26  431  7.27    10.96  565  6.04 
   25.88  324  8.18    14.98  311  4.54 
   21.36  441  9.21    18.31  211  3.77
jaredmauch commented 1 month ago

Looking a bit further as well, it does seem like even some of the included files are not executable as well:

username@usernameiMac QGIS.app % ls -ld Contents/Resources/python/site-packages/cmake/data/CMake.app/Contents/MacOS/CMake 
-rw-r--r--@ 1 username  admin  23743840 Sep 14  2021 Contents/Resources/python/site-packages/cmake/data/CMake.app/Contents/MacOS/CMake
% file Contents/Resources/python/site-packages/cmake/data/CMake.app/Contents/MacOS/CMake   
Contents/Resources/python/site-packages/cmake/data/CMake.app/Contents/MacOS/CMake: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]
Contents/Resources/python/site-packages/cmake/data/CMake.app/Contents/MacOS/CMake (for architecture x86_64):    Mach-O 64-bit executable x86_64
Contents/Resources/python/site-packages/cmake/data/CMake.app/Contents/MacOS/CMake (for architecture arm64): Mach-O 64-bit executable arm64

It almost seems like it's left over from the build environment?