storj-archived / storj-python-sdk

Deprecated. A Python SDK for the Storj API.
https://storj.io
MIT License
23 stars 19 forks source link

use temp file #81

Closed steenzout closed 5 years ago

steenzout commented 7 years ago
    def _make_shards(self):
        """Populates the shard manager with shards."""

        self.shards = []
        self.__postfix = ''
        index = 0

        # Get the file size

        fsize = os.path.getsize(self.filepath)

        optimal_shard_parametrs = \
            self.get_optimal_shard_parametrs(fsize)

        self.__numchunks = int(optimal_shard_parametrs['shard_count'])
        print('Number of chunks %d\n' % self.__numchunks)

        try:
            f = open(self.filepath, 'rb')
        except (OSError, IOError), e:
            raise ShardingException(str(e))

        bname = os.path.split(self.filepath)[1]

        # get chunk size
        self.__chunksize = int(float(fsize) / float(self.__numchunks))

        chunksz = self.__chunksize
        total_bytes = 0
        i = 0
        for x in range(self.__numchunks):
            chunkfilename = bname + '-' + str(x + 1) + self.__postfix

            # if reading the last section,
            # calculate correct chunk size.

            if x == self.__numchunks - 1:
                chunksz = fsize - total_bytes

            self.shard_size = chunksz
            if platform == "linux" or platform == "linux2":
                # linux
                self.tmp_path = '/tmp/'
            elif platform == "darwin":
                # OS X
                self.tmp_path = '/tmp/'
            elif platform == "win32":
                # Windows
                self.tmp_path = "C://Windows/temp/"

            try:
                print('Writing file %s' % chunkfilename)
                data = f.read(chunksz)
                total_bytes += len(data)
                inc = len(data)
                chunkf = file(self.tmp_path + chunkfilename, 'wb')
                chunkf.write(data)
                chunkf.close()
                challenges = self._make_challenges(self.nchallenges)

                shard = Shard(size=self.shard_size, index=index,
                              hash=ShardManager.hash(data),
                              tree=self._make_tree(challenges, data[i:i + inc]),
                              challenges=challenges)
                # hash=ShardManager.hash(data[i:i + inc]),

                # print chunk

                self.shards.append(shard)
                index += 1
                i += 1
            except (OSError, IOError) as e:
                print e
                continue
            except EOFError as e:
                print e
                break

        self.index = len(self.shards)

please assign this to me. I'll create the necessary unit and integration tests to prove the patch will not break functionality.

thank you!

RichardLitt commented 5 years ago

👋 Hey! Thanks for this contribution. Apologies for the delay in responding!

We've decided to rearchitect Storj, so that we can scale better. You can read more about this decision here. This means that we are entirely focused on v3 at the moment, in the storj/storj repository. Our white paper for v3 is coming very, very soon - follow along on the blog and in our Rocketchat.

As this repository is part of the v2 network, we're no longer maintaining this repository. I am going to close this for now. If you have any questions, I encourage you to jump on Rocketchat and ask them there. Thanks!